Can somebody do simply this? By far the strongest engine!

[Ferdy]

There is headsup.exe in release link at https://github.com/fsmosca/HeadsUp

You need the exe and cfg file to be in same folder.

Configure uci engines in headsup.cfg file. Example entries in cfg file.

Code: Select all

[ENGINE1]
# First enginefile must be defined. Do not add quotes even if there is space in the path
enginefile = D:/Chess/Engines/Lc0/lc0-v0.25.1-windows-cpu-openblas/lc0.exe

# Set engine options
WeightsFile = D:/Chess/Engines/Lc0/id/591226.pb.gz
# Threads = 2
MinibatchSize = 8
MaxPrefetch = 0
NNCacheSize = 20000000



[ENGINE2]
# Second enginefile must be defined. Do not add quotes even if there is space in the path
enginefile = D:/Chess/Engines/Stockfish/stockfish_11.exe

# Set engine options
Hash = 128
# Threads = 1
# SyzygyPath = D:/Chess/syzygy
# SyzygyProbeLimit = 5



[HEADSUP OPTION]
# Use engine1 if piece_value_switch and move_number_switch are both satisfied.

# Q=9, R=5, B=3, N=3
# Total piece value excluding kings and pawns = 2 x 9 + 4 x 5 + 4 x 3 + 4 x 3 = 62

# Use engine1 if total piece value remaining is above piece_value_switch.
# Default = 62
piece_value_switch = 0

# Use engine1 if move number is below move_number_switch.
# Default = 0
move_number_switch = 16

# Default = false
Log = false

Important notes:
1. Do not add double quotes for the path of engine or file inside the cfg file
2. piece_value_switch and move_number_switch options need to be all true for the first engine to be used.
* In the current setting, piece_value_switch = 0, says to always use engine1 (except in KvK ending). move_number_switch = 16, says use engine1 from move 1 till move 15. So it will swtich to engine2 only at move 16.
* To use both conditions, like use engine2 if total material remaining on the board is 2Q + 4R or 2*9 + 4*5 or 38 and move number is 30 or more:
piece_value_switch = 38
move_number_switch = 30
If total material balance (except kings and pawns) are still above 38 (where Q=9, R=5 B=N=3) and move number is below 30, then use engine1 otherwise use engine2.

Time forfeit tests are all ok after some games.
TC 15s+0.1s, no time forfeit

Code: Select all

Score of Lc0 0.25.1 blas/Stockfish 11 vs Stockfish 11: 2 - 6 - 8  [0.375] 16
...      Lc0 0.25.1 blas/Stockfish 11 playing White: 2 - 2 - 5  [0.500] 9
...      Lc0 0.25.1 blas/Stockfish 11 playing Black: 0 - 4 - 3  [0.214] 7
...      White vs Black: 6 - 2 - 8  [0.625] 16
Elo difference: -88.7 +/- 126.2, LOS: 7.9 %, DrawRatio: 50.0 %

TC 5s+0.1s, no time forfeit.

Code: Select all

Finished game 10 (Stockfish 11 vs Lc0 0.25.1 blas/Stockfish 11): 1/2-1/2 {Draw by fifty moves rule}
Score of Lc0 0.25.1 blas/Stockfish 11 vs Stockfish 11: 2 - 2 - 8  [0.500] 12
...      Lc0 0.25.1 blas/Stockfish 11 playing White: 2 - 1 - 3  [0.583] 6
...      Lc0 0.25.1 blas/Stockfish 11 playing Black: 0 - 1 - 5  [0.417] 6
...      White vs Black: 3 - 1 - 8  [0.583] 12
Elo difference: 0.0 +/- 117.6, LOS: 50.0 %, DrawRatio: 66.7 %

Sample game at TC 5s+0.1s, Lc0 is only upto move 15.
[pgn]
[Event "?"]
[Site "?"]
[Date "2020.05.18"]
[Round "18"]
[White "Lc0 0.25.1 blas/Stockfish 11"]
[Black "Stockfish 11"]
[Result "1-0"]
[ECO "A40"]
[GameDuration "00:00:21"]
[GameEndTime "2020-05-18T15:34:18.310 Taipei Standard Time"]
[GameStartTime "2020-05-18T15:33:56.449 Taipei Standard Time"]
[Opening "Modern defense"]
[PlyCount "137"]
[TimeControl "5+0.1"]

1. d4 {book} g6 {book} 2. e4 {+0.33/3 0.20s} d6 {-0.56/16 0.37s}
3. c4 {+0.35/4 0.20s} Bg7 {-0.15/12 0.079s} 4. Nc3 {+0.35/4 0.20s}
Nf6 {-0.50/15 0.32s} 5. Be2 {+0.37/5 0.20s} O-O {-0.43/15 0.13s}
6. Nf3 {+0.34/5 0.20s} e5 {-0.34/16 0.31s} 7. O-O {+0.33/5 0.20s}
exd4 {-0.37/14 0.093s} 8. Nxd4 {+0.31/6 0.20s} Nc6 {-0.36/17 0.22s}
9. Be3 {+0.40/5 0.20s} Re8 {-0.35/17 0.29s} 10. f3 {+0.39/5 0.20s}
Nxd4 {-0.45/14 0.080s} 11. Bxd4 {+0.44/5 0.20s} Bh6 {-0.54/15 0.13s}
12. Kh1 {+0.37/2 0.20s} Nh5 {-0.03/17 0.43s} 13. g3 {+0.36/3 0.20s}
Bh3 {-0.22/14 0.17s} 14. Rg1 {+0.37/3 0.20s} Be6 {-0.38/16 0.45s}
15. f4 {+0.45/3 0.20s} Nf6 {-0.36/13 0.085s} 16. Qd3 {+0.95/15 0.23s}
Bg4 {-0.33/13 0.094s} 17. Bd1 {+1.09/16 0.28s} Bxd1 {-0.68/18 0.88s}
18. Raxd1 {+1.07/15 0.078s} Bg7 {-0.76/13 0.099s} 19. Rgf1 {+1.49/16 0.37s}
Qe7 {-0.91/16 0.93s} 20. f5 {+1.91/16 0.23s} c6 {-0.63/13 0.064s}
21. Qf3 {+1.59/16 0.26s} g5 {-0.63/14 0.13s} 22. h3 {+1.38/18 0.75s}
h6 {-0.67/14 0.27s} 23. Kg2 {+1.29/16 0.38s} Rad8 {-0.67/12 0.090s}
24. Rde1 {+1.41/14 0.12s} a6 {-0.70/17 0.40s} 25. a4 {+1.26/15 0.11s}
Qc7 {-0.89/15 0.28s} 26. h4 {+1.27/16 0.54s} gxh4 {-0.69/15 0.19s}
27. Rh1 {+2.13/13 0.053s} d5 {-0.81/13 0.038s} 28. e5 {+2.49/15 0.16s}
Nh7 {-0.76/17 0.28s} 29. Qg4 {+1.98/20 0.52s} dxc4 {-0.77/15 0.057s}
30. f6 {+2.01/15 0.043s} h3+ {-1.86/17 0.38s} 31. Rxh3 {+2.94/16 0.065s}
Ng5 {-1.84/18 0.099s} 32. fxg7 {+4.19/16 0.068s} Nxh3 {-1.69/16 0.085s}
33. Ne4 {+6.66/14 0.068s} Ng5 {-5.35/19 0.35s} 34. Nf6+ {+7.67/19 0.15s}
Kxg7 {-5.35/1 0s} 35. e6 {+7.87/15 0.073s} Rxe6 {-7.22/16 0.27s}
36. Rxe6 {+7.92/16 0.12s} Qa5 {-5.25/14 0.069s} 37. Re2 {+8.61/15 0.067s}
Rxd4 {-7.26/17 0.18s} 38. Qxd4 {+8.85/13 0.069s} Ne6 {-7.80/15 0.090s}
39. Ne8+ {+8.33/18 0.24s} Kf8 {-8.93/15 0.13s} 40. Qh8+ {+8.92/15 0.081s}
Ke7 {-8.93/1 0s} 41. Nf6 {+8.98/16 0.073s} h5 {-8.24/13 0.059s}
42. Rf2 {+9.14/17 0.23s} Nf8 {-8.15/14 0.10s} 43. Ng8+ {+9.06/15 0.17s}
Ke8 {-7.77/14 0.070s} 44. Nh6 {+9.13/15 0.063s} Qd5+ {-6.93/12 0.044s}
45. Kh2 {+9.21/16 0.11s} h4 {-8.92/14 0.26s} 46. Nf5 {+9.32/16 0.10s}
hxg3+ {-9.22/14 0.16s} 47. Kxg3 {+9.45/15 0.074s} Qd1 {-8.59/15 0.11s}
48. a5 {+9.38/14 0.18s} Qe1 {-9.97/13 0.10s} 49. Kf3 {+10.54/13 0.044s}
Qd1+ {-12.00/14 0.099s} 50. Re2+ {+12.93/17 0.091s} Kd7 {-12.97/17 0.074s}
51. Qxf8 {+14.86/19 0.16s} Qf1+ {-14.62/17 0.13s} 52. Kg3 {+17.00/15 0.045s}
Qg1+ {-16.94/13 0.10s} 53. Kh4 {+17.78/17 0.10s} Qh1+ {-17.37/13 0.097s}
54. Kg4 {+17.99/16 0.073s} Qg1+ {-17.41/17 0.10s} 55. Ng3 {+18.83/17 0.095s}
Qd1 {-10.35/13 0.068s} 56. Kh3 {+23.51/16 0.071s} Qxe2 {-17.56/14 0.13s}
57. Nxe2 {+28.49/18 0.098s} Ke6 {-27.21/15 0.10s} 58. Qc8+ {+29.02/15 0.073s}
Kd6 {-61.11/17 0.10s} 59. Qxb7 {+61.47/18 0.094s} c3 {-60.23/17 0.099s}
60. Nxc3 {+65.00/18 0.067s} f5 {-65.42/16 0.081s} 61. Qxa6 {+71.24/18 0.073s}
f4 {-M38/17 0.12s} 62. Kg4 {+319.86/17 0.080s} f3 {-M22/16 0.099s}
63. Kxf3 {+319.91/19 0.070s} Kc7 {-M14/18 0.032s} 64. Ne4 {+319.93/21 0.074s}
Kd7 {-M12/22 0.039s} 65. Qb7+ {+319.94/24 0.071s} Ke6 {-M10/24 0.045s}
66. Kf4 {+319.96/36 0.12s} c5 {-M6/130 0.045s} 67. Ng5+ {+319.97/245 0.072s}
Kd6 {-M4/245 0.012s} 68. Kf5 {+319.98/245 0.040s} c4 {-M2/1 0s}
69. Ne4# {+319.99/245 0.032s, White mates} 1-0
[/pgn]

[Ferdy]

Strength comparison between Lc0 0.25.1 blas and Ethereal 12.00 for up to move 15. TC 5s+0.1s, no ET and both are using 2 threads from 1 and 2 plies opening.

Code: Select all

Score of Lc0 0.25.1 blas/Stockfish 11 vs Stockfish 11: 19 - 26 - 55  [0.465] 100
...      Lc0 0.25.1 blas/Stockfish 11 playing White: 11 - 11 - 28  [0.500] 50
...      Lc0 0.25.1 blas/Stockfish 11 playing Black: 8 - 15 - 27  [0.430] 50
...      White vs Black: 26 - 19 - 55  [0.535] 100
Elo difference: -24.4 +/- 45.9, LOS: 14.8 %, DrawRatio: 55.0 %

Code: Select all

Score of Ethereal 12.00/Stockfish 11 vs Stockfish 11: 14 - 36 - 50  [0.390] 100
...      Ethereal 12.00/Stockfish 11 playing White: 12 - 9 - 29  [0.530] 50
...      Ethereal 12.00/Stockfish 11 playing Black: 2 - 27 - 21  [0.250] 50
...      White vs Black: 39 - 11 - 50  [0.640] 100
Elo difference: -77.7 +/- 48.4, LOS: 0.1 %, DrawRatio: 50.0 %

Lc0 0.25.1 blas performed better by around -24.4 - (-77.7) or 53 rating points with common opponent Stockfish 11.

[pohl4711]

Also Eman Chimera can change engine by number of pieces on the board not just move number.

How ????

In the latest version 1.1 I found no way to do so. Only movenumber work.

[Laskos]

There is headsup.exe in release link at https://github.com/fsmosca/HeadsUp

You need the exe and cfg file to be in same folder.

Configure uci engines in headsup.cfg file. Example entries in cfg file.

Code: Select all

[ENGINE1]
# First enginefile must be defined. Do not add quotes even if there is space in the path
enginefile = D:/Chess/Engines/Lc0/lc0-v0.25.1-windows-cpu-openblas/lc0.exe

# Set engine options
WeightsFile = D:/Chess/Engines/Lc0/id/591226.pb.gz
# Threads = 2
MinibatchSize = 8
MaxPrefetch = 0
NNCacheSize = 20000000



[ENGINE2]
# Second enginefile must be defined. Do not add quotes even if there is space in the path
enginefile = D:/Chess/Engines/Stockfish/stockfish_11.exe

# Set engine options
Hash = 128
# Threads = 1
# SyzygyPath = D:/Chess/syzygy
# SyzygyProbeLimit = 5



[HEADSUP OPTION]
# Use engine1 if piece_value_switch and move_number_switch are both satisfied.

# Q=9, R=5, B=3, N=3
# Total piece value excluding kings and pawns = 2 x 9 + 4 x 5 + 4 x 3 + 4 x 3 = 62

# Use engine1 if total piece value remaining is above piece_value_switch.
# Default = 62
piece_value_switch = 0

# Use engine1 if move number is below move_number_switch.
# Default = 0
move_number_switch = 16

# Default = false
Log = false

Important notes:
1. Do not add double quotes for the path of engine or file inside the cfg file
2. piece_value_switch and move_number_switch options need to be all true for the first engine to be used.
* In the current setting, piece_value_switch = 0, says to always use engine1 (except in KvK ending). move_number_switch = 16, says use engine1 from move 1 till move 15. So it will swtich to engine2 only at move 16.
* To use both conditions, like use engine2 if total material remaining on the board is 2Q + 4R or 2*9 + 4*5 or 38 and move number is 30 or more:
piece_value_switch = 38
move_number_switch = 30
If total material balance (except kings and pawns) are still above 38 (where Q=9, R=5 B=N=3) and move number is below 30, then use engine1 otherwise use engine2.

Time forfeit tests are all ok after some games.
TC 15s+0.1s, no time forfeit

Code: Select all

Score of Lc0 0.25.1 blas/Stockfish 11 vs Stockfish 11: 2 - 6 - 8  [0.375] 16
...      Lc0 0.25.1 blas/Stockfish 11 playing White: 2 - 2 - 5  [0.500] 9
...      Lc0 0.25.1 blas/Stockfish 11 playing Black: 0 - 4 - 3  [0.214] 7
...      White vs Black: 6 - 2 - 8  [0.625] 16
Elo difference: -88.7 +/- 126.2, LOS: 7.9 %, DrawRatio: 50.0 %

TC 5s+0.1s, no time forfeit.

Code: Select all

Finished game 10 (Stockfish 11 vs Lc0 0.25.1 blas/Stockfish 11): 1/2-1/2 {Draw by fifty moves rule}
Score of Lc0 0.25.1 blas/Stockfish 11 vs Stockfish 11: 2 - 2 - 8  [0.500] 12
...      Lc0 0.25.1 blas/Stockfish 11 playing White: 2 - 1 - 3  [0.583] 6
...      Lc0 0.25.1 blas/Stockfish 11 playing Black: 0 - 1 - 5  [0.417] 6
...      White vs Black: 3 - 1 - 8  [0.583] 12
Elo difference: 0.0 +/- 117.6, LOS: 50.0 %, DrawRatio: 66.7 %

Sample game at TC 5s+0.1s, Lc0 is only upto move 15.
[pgn]
[Event "?"]
[Site "?"]
[Date "2020.05.18"]
[Round "18"]
[White "Lc0 0.25.1 blas/Stockfish 11"]
[Black "Stockfish 11"]
[Result "1-0"]
[ECO "A40"]
[GameDuration "00:00:21"]
[GameEndTime "2020-05-18T15:34:18.310 Taipei Standard Time"]
[GameStartTime "2020-05-18T15:33:56.449 Taipei Standard Time"]
[Opening "Modern defense"]
[PlyCount "137"]
[TimeControl "5+0.1"]

1. d4 {book} g6 {book} 2. e4 {+0.33/3 0.20s} d6 {-0.56/16 0.37s}
3. c4 {+0.35/4 0.20s} Bg7 {-0.15/12 0.079s} 4. Nc3 {+0.35/4 0.20s}
Nf6 {-0.50/15 0.32s} 5. Be2 {+0.37/5 0.20s} O-O {-0.43/15 0.13s}
6. Nf3 {+0.34/5 0.20s} e5 {-0.34/16 0.31s} 7. O-O {+0.33/5 0.20s}
exd4 {-0.37/14 0.093s} 8. Nxd4 {+0.31/6 0.20s} Nc6 {-0.36/17 0.22s}
9. Be3 {+0.40/5 0.20s} Re8 {-0.35/17 0.29s} 10. f3 {+0.39/5 0.20s}
Nxd4 {-0.45/14 0.080s} 11. Bxd4 {+0.44/5 0.20s} Bh6 {-0.54/15 0.13s}
12. Kh1 {+0.37/2 0.20s} Nh5 {-0.03/17 0.43s} 13. g3 {+0.36/3 0.20s}
Bh3 {-0.22/14 0.17s} 14. Rg1 {+0.37/3 0.20s} Be6 {-0.38/16 0.45s}
15. f4 {+0.45/3 0.20s} Nf6 {-0.36/13 0.085s} 16. Qd3 {+0.95/15 0.23s}
Bg4 {-0.33/13 0.094s} 17. Bd1 {+1.09/16 0.28s} Bxd1 {-0.68/18 0.88s}
18. Raxd1 {+1.07/15 0.078s} Bg7 {-0.76/13 0.099s} 19. Rgf1 {+1.49/16 0.37s}
Qe7 {-0.91/16 0.93s} 20. f5 {+1.91/16 0.23s} c6 {-0.63/13 0.064s}
21. Qf3 {+1.59/16 0.26s} g5 {-0.63/14 0.13s} 22. h3 {+1.38/18 0.75s}
h6 {-0.67/14 0.27s} 23. Kg2 {+1.29/16 0.38s} Rad8 {-0.67/12 0.090s}
24. Rde1 {+1.41/14 0.12s} a6 {-0.70/17 0.40s} 25. a4 {+1.26/15 0.11s}
Qc7 {-0.89/15 0.28s} 26. h4 {+1.27/16 0.54s} gxh4 {-0.69/15 0.19s}
27. Rh1 {+2.13/13 0.053s} d5 {-0.81/13 0.038s} 28. e5 {+2.49/15 0.16s}
Nh7 {-0.76/17 0.28s} 29. Qg4 {+1.98/20 0.52s} dxc4 {-0.77/15 0.057s}
30. f6 {+2.01/15 0.043s} h3+ {-1.86/17 0.38s} 31. Rxh3 {+2.94/16 0.065s}
Ng5 {-1.84/18 0.099s} 32. fxg7 {+4.19/16 0.068s} Nxh3 {-1.69/16 0.085s}
33. Ne4 {+6.66/14 0.068s} Ng5 {-5.35/19 0.35s} 34. Nf6+ {+7.67/19 0.15s}
Kxg7 {-5.35/1 0s} 35. e6 {+7.87/15 0.073s} Rxe6 {-7.22/16 0.27s}
36. Rxe6 {+7.92/16 0.12s} Qa5 {-5.25/14 0.069s} 37. Re2 {+8.61/15 0.067s}
Rxd4 {-7.26/17 0.18s} 38. Qxd4 {+8.85/13 0.069s} Ne6 {-7.80/15 0.090s}
39. Ne8+ {+8.33/18 0.24s} Kf8 {-8.93/15 0.13s} 40. Qh8+ {+8.92/15 0.081s}
Ke7 {-8.93/1 0s} 41. Nf6 {+8.98/16 0.073s} h5 {-8.24/13 0.059s}
42. Rf2 {+9.14/17 0.23s} Nf8 {-8.15/14 0.10s} 43. Ng8+ {+9.06/15 0.17s}
Ke8 {-7.77/14 0.070s} 44. Nh6 {+9.13/15 0.063s} Qd5+ {-6.93/12 0.044s}
45. Kh2 {+9.21/16 0.11s} h4 {-8.92/14 0.26s} 46. Nf5 {+9.32/16 0.10s}
hxg3+ {-9.22/14 0.16s} 47. Kxg3 {+9.45/15 0.074s} Qd1 {-8.59/15 0.11s}
48. a5 {+9.38/14 0.18s} Qe1 {-9.97/13 0.10s} 49. Kf3 {+10.54/13 0.044s}
Qd1+ {-12.00/14 0.099s} 50. Re2+ {+12.93/17 0.091s} Kd7 {-12.97/17 0.074s}
51. Qxf8 {+14.86/19 0.16s} Qf1+ {-14.62/17 0.13s} 52. Kg3 {+17.00/15 0.045s}
Qg1+ {-16.94/13 0.10s} 53. Kh4 {+17.78/17 0.10s} Qh1+ {-17.37/13 0.097s}
54. Kg4 {+17.99/16 0.073s} Qg1+ {-17.41/17 0.10s} 55. Ng3 {+18.83/17 0.095s}
Qd1 {-10.35/13 0.068s} 56. Kh3 {+23.51/16 0.071s} Qxe2 {-17.56/14 0.13s}
57. Nxe2 {+28.49/18 0.098s} Ke6 {-27.21/15 0.10s} 58. Qc8+ {+29.02/15 0.073s}
Kd6 {-61.11/17 0.10s} 59. Qxb7 {+61.47/18 0.094s} c3 {-60.23/17 0.099s}
60. Nxc3 {+65.00/18 0.067s} f5 {-65.42/16 0.081s} 61. Qxa6 {+71.24/18 0.073s}
f4 {-M38/17 0.12s} 62. Kg4 {+319.86/17 0.080s} f3 {-M22/16 0.099s}
63. Kxf3 {+319.91/19 0.070s} Kc7 {-M14/18 0.032s} 64. Ne4 {+319.93/21 0.074s}
Kd7 {-M12/22 0.039s} 65. Qb7+ {+319.94/24 0.071s} Ke6 {-M10/24 0.045s}
66. Kf4 {+319.96/36 0.12s} c5 {-M6/130 0.045s} 67. Ng5+ {+319.97/245 0.072s}
Kd6 {-M4/245 0.012s} 68. Kf5 {+319.98/245 0.040s} c4 {-M2/1 0s}
69. Ne4# {+319.99/245 0.032s, White mates} 1-0
[/pgn]

I for some reason get "disconnect" messages in Cutechess-CLI and all games are lost for headsup.

Code: Select all

Terminating process of engine headsup_24(0)
Finished game 1 (headsup_24 vs SF_11): 0-1 {White disconnects}
Score of headsup_24 vs SF_11: 0 - 1 - 0  [0.000] 1
Elo difference: -inf +/- nan
Terminating process of engine headsup_24(2)
Finished game 2 (SF_11 vs headsup_24): 1-0 {Black disconnects}
Score of headsup_24 vs SF_11: 0 - 2 - 0  [0.000] 2
Elo difference: -inf +/- nan
Terminating process of engine headsup_24(3)
Finished game 3 (headsup_24 vs SF_11): 0-1 {White disconnects}
Score of headsup_24 vs SF_11: 0 - 3 - 0  [0.000] 3
Elo difference: -inf +/- nan

Here is my cfg file:

Code: Select all

[ENGINE1]
# First enginefile must be defined. Do not add quotes even if there is space in the path
enginefile = C:/Users/Kai/Downloads/LC0_CUDA_3/lc0_v251.exe

# Set engine options
WeightsFile = C:/Users/Kai/Downloads/LC0_CUDA_3/591226.pb.gz
# Threads = 2
NNCacheSize = 500000
MoveOverheadMs = 0



[ENGINE2]
# Second enginefile must be defined. Do not add quotes even if there is space in the path
enginefile = C:/Users/Kai/Downloads/LC0_CUDA_3/stockfish_11.exe

# Set engine options
Hash = 128
Threads = 4
Move Overhead = 0
# SyzygyPath = D:/Chess/syzygy
# SyzygyProbeLimit = 5



[HEADSUP OPTION]
# Use engine1 if piece_value_switch and move_number_switch are both satisfied.

# Q=9, R=5, B=3, N=3
# Total piece value excluding kings and pawns = 2 x 9 + 4 x 5 + 4 x 3 + 4 x 3 = 62

# Use engine1 if total piece value remaining is above piece_value_switch.
# Default = 62
piece_value_switch = 24

# Use engine1 if move number is below move_number_switch.
# Default = 0
move_number_switch = 0

# Default = false
Log = true

Here is the log file:

Code: Select all

2020-05-18 15:55:19,233 :: DEBUG :: Using proactor: IocpProactor
2020-05-18 15:55:19,243 :: DEBUG :: <UciProtocol (pid=6464)>: Connection made
2020-05-18 15:55:19,243 :: DEBUG :: <UciProtocol (pid=6464)>: << uci
2020-05-18 15:55:20,624 :: WARNING :: <UciProtocol (pid=6464)>: stderr >>        _
2020-05-18 15:55:20,625 :: WARNING :: <UciProtocol (pid=6464)>: stderr >> |   _ | |
2020-05-18 15:55:20,625 :: WARNING :: <UciProtocol (pid=6464)>: stderr >> |_ |_ |_| v0.25.1+git.69105b4 built Apr 30 2020
2020-05-18 15:55:20,627 :: DEBUG :: <UciProtocol (pid=6464)>: >> id name Lc0 v0.25.1+git.69105b4
2020-05-18 15:55:20,628 :: DEBUG :: <UciProtocol (pid=6464)>: >> id author The LCZero Authors.
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name WeightsFile type string default <autodiscover>
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name Backend type combo default cudnn-auto var cudnn-auto var cudnn var cudnn-fp16 var random var check var roundrobin var multiplexing var demux
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name BackendOptions type string default 
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name Threads type spin default 2 min 1 max 128
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name NNCacheSize type spin default 200000 min 0 max 999999999
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MinibatchSize type spin default 256 min 1 max 1024
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MaxPrefetch type spin default 32 min 0 max 1024
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name LogitQ type check default false
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name CPuct type string default 2.147000
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name CPuctAtRoot type string default 2.147000
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name CPuctBase type string default 18368.000000
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name CPuctBaseAtRoot type string default 18368.000000
2020-05-18 15:55:20,629 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name CPuctFactor type string default 2.815000
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name CPuctFactorAtRoot type string default 2.815000
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name VerboseMoveStats type check default false
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name FpuStrategy type combo default reduction var reduction var absolute
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name FpuValue type string default 0.443000
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name FpuStrategyAtRoot type combo default same var reduction var absolute var same
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name FpuValueAtRoot type string default 1.000000
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name CacheHistoryLength type spin default 0 min 0 max 7
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name PolicyTemperature type string default 1.607000
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MaxCollisionEvents type spin default 32 min 1 max 65536
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MaxCollisionVisits type spin default 9999 min 1 max 1000000
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name OutOfOrderEval type check default true
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MaxOutOfOrderEvalsFactor type string default 1.000000
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name StickyEndgames type check default true
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name SyzygyFastPlay type check default true
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MultiPV type spin default 1 min 1 max 500
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name PerPVCounters type check default false
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name ScoreType type combo default centipawn var centipawn var centipawn_with_drawscore var centipawn_2019 var centipawn_2018 var win_percentage var Q var W-L
2020-05-18 15:55:20,630 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name HistoryFill type combo default fen_only var no var fen_only var always
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MovesLeftMaxEffect type string default 0.000000
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MovesLeftThreshold type string default 1.000000
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MovesLeftSlope type string default 0.001000
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name ShortSightedness type string default 0.000000
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MaxConcurrentSearchers type spin default 1 min 0 max 128
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name DrawScoreSideToMove type spin default 0 min -100 max 100
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name DrawScoreOpponent type spin default 0 min -100 max 100
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name DrawScoreWhite type spin default 0 min -100 max 100
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name DrawScoreBlack type spin default 0 min -100 max 100
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name SyzygyPath type string default 
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name Ponder type check default true
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name UCI_Chess960 type check default false
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name UCI_ShowWDL type check default false
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name ConfigFile type string default lc0.config
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name KLDGainAverageInterval type spin default 100 min 1 max 10000000
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MinimumKLDGainPerNode type string default 0.000000
2020-05-18 15:55:20,631 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name SmartPruningFactor type string default 1.330000
2020-05-18 15:55:20,632 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name SmartPruningMinimumBatches type spin default 0 min 0 max 10000
2020-05-18 15:55:20,632 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name RamLimitMb type spin default 0 min 0 max 100000000
2020-05-18 15:55:20,632 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name MoveOverheadMs type spin default 200 min 0 max 100000000
2020-05-18 15:55:20,632 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name TimeManager type string default legacy
2020-05-18 15:55:20,632 :: DEBUG :: <UciProtocol (pid=6464)>: >> option name LogFile type string default 
2020-05-18 15:55:20,632 :: DEBUG :: <UciProtocol (pid=6464)>: >> uciok
2020-05-18 15:55:20,632 :: DEBUG :: Using proactor: IocpProactor
2020-05-18 15:55:20,640 :: DEBUG :: <UciProtocol (pid=13816)>: Connection made
2020-05-18 15:55:20,641 :: DEBUG :: <UciProtocol (pid=13816)>: << uci
2020-05-18 15:55:20,642 :: DEBUG :: <UciProtocol (pid=13816)>: >> Stockfish 11 64 BMI2 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> id name Stockfish 11 64 BMI2
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> id author T. Romstad, M. Costalba, J. Kiiski, G. Linscott
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> 
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Debug Log File type string default 
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Contempt type spin default 24 min -100 max 100
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Analysis Contempt type combo default Both var Off var White var Black var Both
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Threads type spin default 1 min 1 max 512
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Hash type spin default 16 min 1 max 131072
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Clear Hash type button
2020-05-18 15:55:20,674 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Ponder type check default false
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name MultiPV type spin default 1 min 1 max 500
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Skill Level type spin default 20 min 0 max 20
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Move Overhead type spin default 30 min 0 max 5000
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Minimum Thinking Time type spin default 20 min 0 max 5000
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Slow Mover type spin default 84 min 10 max 1000
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name nodestime type spin default 0 min 0 max 10000
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name UCI_Chess960 type check default false
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name UCI_AnalyseMode type check default false
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name UCI_LimitStrength type check default false
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name UCI_Elo type spin default 1350 min 1350 max 2850
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name SyzygyPath type string default <empty>
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name SyzygyProbeDepth type spin default 1 min 1 max 100
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name Syzygy50MoveRule type check default true
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> option name SyzygyProbeLimit type spin default 7 min 0 max 7
2020-05-18 15:55:20,675 :: DEBUG :: <UciProtocol (pid=13816)>: >> uciok
2020-05-18 15:55:20,677 :: DEBUG :: <UciProtocol (pid=6464)>: << setoption name weightsfile value C:/Users/Kai/Downloads/LC0_CUDA_3/591226.pb.gz
2020-05-18 15:55:20,678 :: DEBUG :: <UciProtocol (pid=6464)>: << setoption name nncachesize value 500000
2020-05-18 15:55:20,678 :: DEBUG :: <UciProtocol (pid=6464)>: << setoption name moveoverheadms value 0
2020-05-18 15:55:20,679 :: DEBUG :: <UciProtocol (pid=13816)>: << setoption name hash value 128
2020-05-18 15:55:20,679 :: DEBUG :: <UciProtocol (pid=13816)>: << setoption name threads value 4
2020-05-18 15:55:20,680 :: DEBUG :: <UciProtocol (pid=13816)>: << setoption name move overhead value 0

[Ferdy]

My batch file to run the match is like this.

Code: Select all

set CCC=D:\Chess\CuteChess-CLI\cutechess-cli.exe

%CCC% --help >help.txt
%CCC% --version

set CON=3
set E1="HeadsUp2"
set E2="Stockfish 11"

set STARTOPENING=docs\Noomen_3move.pgn
set OPEFORMAT=pgn

set SAVE=games_tc15s100ms.pgn

%CCC% ^
-tournament round-robin ^
-rounds 500 ^
-games 2 ^
-repeat ^
-openings file=%STARTOPENING% format=%OPEFORMAT% order=random plies=1000 start=1 policy=round ^
-concurrency %CON% ^
-pgnout %SAVE% fi ^
-wait 100 ^
-maxmoves 500 ^
-ratinginterval 4 ^
-engine conf=%E1% tc=0/15+0.1 stderr=err1.txt name="Lc0 v0.25.1/Stockfish 11" ^
-engine conf=%E2% tc=0/15+0.1 stderr=err2.txt

"HeadsUp2" and "Stockfish 11" are registered in engine json file.

I would like you to add the stderr option like in the following:

Code: Select all

-engine conf=%E1% tc=0/15+0.1 stderr=err1.txt name="Lc0 v0.25.1/Stockfish 11"
-engine conf=%E2% tc=0/15+0.1 stderr=err2.txt

Then let's see what are in err1.txt and err2.txt files.

[khalidomar]

The switch should be done based on material left on board and mobility of material left on board (and/or number of free pawns), not simply by move number. This could even be nicely auto-tuned for each TC.

Eman Chimera already supports switching engines based on (a) move number or (b) remaining piece count

[khalidomar]

How ????

In the latest version 1.1 I found no way to do so. Only movenumber work.

Download v2.00 from: https://eman.zohosites.com/eman-chimera.html

[Laskos]

My batch file to run the match is like this.

Code: Select all

set CCC=D:\Chess\CuteChess-CLI\cutechess-cli.exe

%CCC% --help >help.txt
%CCC% --version

set CON=3
set E1="HeadsUp2"
set E2="Stockfish 11"

set STARTOPENING=docs\Noomen_3move.pgn
set OPEFORMAT=pgn

set SAVE=games_tc15s100ms.pgn

%CCC% ^
-tournament round-robin ^
-rounds 500 ^
-games 2 ^
-repeat ^
-openings file=%STARTOPENING% format=%OPEFORMAT% order=random plies=1000 start=1 policy=round ^
-concurrency %CON% ^
-pgnout %SAVE% fi ^
-wait 100 ^
-maxmoves 500 ^
-ratinginterval 4 ^
-engine conf=%E1% tc=0/15+0.1 stderr=err1.txt name="Lc0 v0.25.1/Stockfish 11" ^
-engine conf=%E2% tc=0/15+0.1 stderr=err2.txt

"HeadsUp2" and "Stockfish 11" are registered in engine json file.

I would like you to add the stderr option like in the following:

Code: Select all

-engine conf=%E1% tc=0/15+0.1 stderr=err1.txt name="Lc0 v0.25.1/Stockfish 11"
-engine conf=%E2% tc=0/15+0.1 stderr=err2.txt

Then let's see what are in err1.txt and err2.txt files.

Sorry, for delay, I will now look into it. Whas curious what Chimera can bring.

[Laskos]

My batch file to run the match is like this.

Code: Select all

set CCC=D:\Chess\CuteChess-CLI\cutechess-cli.exe

%CCC% --help >help.txt
%CCC% --version

set CON=3
set E1="HeadsUp2"
set E2="Stockfish 11"

set STARTOPENING=docs\Noomen_3move.pgn
set OPEFORMAT=pgn

set SAVE=games_tc15s100ms.pgn

%CCC% ^
-tournament round-robin ^
-rounds 500 ^
-games 2 ^
-repeat ^
-openings file=%STARTOPENING% format=%OPEFORMAT% order=random plies=1000 start=1 policy=round ^
-concurrency %CON% ^
-pgnout %SAVE% fi ^
-wait 100 ^
-maxmoves 500 ^
-ratinginterval 4 ^
-engine conf=%E1% tc=0/15+0.1 stderr=err1.txt name="Lc0 v0.25.1/Stockfish 11" ^
-engine conf=%E2% tc=0/15+0.1 stderr=err2.txt

"HeadsUp2" and "Stockfish 11" are registered in engine json file.

I would like you to add the stderr option like in the following:

Code: Select all

-engine conf=%E1% tc=0/15+0.1 stderr=err1.txt name="Lc0 v0.25.1/Stockfish 11"
-engine conf=%E2% tc=0/15+0.1 stderr=err2.txt

Then let's see what are in err1.txt and err2.txt files.

Ferdy, sorry for the delay, I seem to not get anything in the "err" files (empty). Cutechess-CLI indexes the opening suite (2moves_v1.pgn) starts the first game then "disconnects" and ends. The PGN file reads as:

Code: Select all

[Event "?"]
[Site "?"]
[Date "2020.05.20"]
[Round "1"]
[White "Lc0 v0.25.1/Stockfish 11"]
[Black "SF_11"]
[Result "0-1"]
[FEN "r1bqkbnr/pppppp1p/n7/6p1/6P1/7P/PPPPPP2/RNBQKBNR w KQkq - 0 1"]
[GameDuration "00:00:00"]
[GameEndTime "2020-05-20T13:13:21.268 GTB Daylight Time"]
[PlyCount "0"]
[SetUp "1"]
[Termination "abandoned"]
[TimeControl "15+0.1"]

{White disconnects} 0-1

"headsup" engine seems to run and after typing "UCI" it shows:

Code: Select all

uci
id name HeadsUp 0.1
id author Ferdy
info string engine1 is Lc0 v0.25.1+git.69105b4
info string engine2 is Stockfish 11 64 BMI2
uciok

[Ferdy]

My batch file to run the match is like this.

Code: Select all

set CCC=D:\Chess\CuteChess-CLI\cutechess-cli.exe

%CCC% --help >help.txt
%CCC% --version

set CON=3
set E1="HeadsUp2"
set E2="Stockfish 11"

set STARTOPENING=docs\Noomen_3move.pgn
set OPEFORMAT=pgn

set SAVE=games_tc15s100ms.pgn

%CCC% ^
-tournament round-robin ^
-rounds 500 ^
-games 2 ^
-repeat ^
-openings file=%STARTOPENING% format=%OPEFORMAT% order=random plies=1000 start=1 policy=round ^
-concurrency %CON% ^
-pgnout %SAVE% fi ^
-wait 100 ^
-maxmoves 500 ^
-ratinginterval 4 ^
-engine conf=%E1% tc=0/15+0.1 stderr=err1.txt name="Lc0 v0.25.1/Stockfish 11" ^
-engine conf=%E2% tc=0/15+0.1 stderr=err2.txt

"HeadsUp2" and "Stockfish 11" are registered in engine json file.

I would like you to add the stderr option like in the following:

Code: Select all

-engine conf=%E1% tc=0/15+0.1 stderr=err1.txt name="Lc0 v0.25.1/Stockfish 11"
-engine conf=%E2% tc=0/15+0.1 stderr=err2.txt

Then let's see what are in err1.txt and err2.txt files.

Ferdy, sorry for the delay, I seem to not get anything in the "err" files (empty). Cutechess-CLI indexes the opening suite (2moves_v1.pgn) starts the first game then "disconnects" and ends. The PGN file reads as:

Code: Select all

[Event "?"]
[Site "?"]
[Date "2020.05.20"]
[Round "1"]
[White "Lc0 v0.25.1/Stockfish 11"]
[Black "SF_11"]
[Result "0-1"]
[FEN "r1bqkbnr/pppppp1p/n7/6p1/6P1/7P/PPPPPP2/RNBQKBNR w KQkq - 0 1"]
[GameDuration "00:00:00"]
[GameEndTime "2020-05-20T13:13:21.268 GTB Daylight Time"]
[PlyCount "0"]
[SetUp "1"]
[Termination "abandoned"]
[TimeControl "15+0.1"]

{White disconnects} 0-1

"headsup" engine seems to run and after typing "UCI" it shows:

Code: Select all

uci
id name HeadsUp 0.1
id author Ferdy
info string engine1 is Lc0 v0.25.1+git.69105b4
info string engine2 is Stockfish 11 64 BMI2
uciok

Lets see some more logs.
To see the logs of engine itself, add the following options to each engine in the cfg file.
For Lc0

Code: Select all

LogFile = log_lc0.txt

For Stockfish

Code: Select all

Debug Log File = log_sf.txt

or something like this.

Code: Select all

[ENGINE1]
# First enginefile must be defined. Do not add quotes even if there is space in the path
enginefile = D:/Chess/Engines/Lc0/lc0-v0.25.1-windows-cpu-openblas/lc0.exe

# Set engine options
WeightsFile = D:/Chess/Engines/Lc0/id/591226.pb.gz
# Threads = 2
MinibatchSize = 8
MaxPrefetch = 0
NNCacheSize = 20000000
LogFile = log_lc0.txt



[ENGINE2]
# Second enginefile must be defined. Do not add quotes even if there is space in the path
enginefile = D:/Chess/Engines/Stockfish/stockfish_11.exe

# Set engine options
Hash = 128
# Threads = 1
# SyzygyPath = D:/Chess/syzygy
# SyzygyProbeLimit = 5
Debug Log File = log_sf.txt



[HEADSUP OPTION]
# Use engine1 if piece_value_switch and move_number_switch are both satisfied.

# Q=9, R=5, B=3, N=3
# Total piece value excluding kings and pawns = 2 x 9 + 4 x 5 + 4 x 3 + 4 x 3 = 62

# Use engine1 if total piece value remaining is above piece_value_switch.
# Default = 62
piece_value_switch = 24

# Use engine1 if move number is below move_number_switch.
# Default = 0
move_number_switch = 0

# Default = false
Log = true

Note that the settings,

Code: Select all

piece_value_switch = 24
move_number_switch = 0

only activates the 2nd engine or Stockfish in that case.

So lets look the contents of log_sf.txt. This file can be found where the HeadsUp.exe is located.

To see the logging of cutechess-cli itself modify the command line to have -debug option

Code: Select all

cutechess-cli ... -debug >log_cutechess.txt ...

I did try the file 2moves_v1.pgn and HeadsUp has no problem with it.

Note I am using cutechess-cli and cutechess-gui from here.