buying a new computer

[corres]

In general if one uses two systems like an AB engine and an NN engine for making analysis that one should be a good chess player because only that one should choose the right move if there is a difference in the value of the analyzed position between the two engines.
So for a weaker chess player it is better to use only one - AB or NN - system than two.
Maybe a cross analysis can help in this situation too but it is a very time consuming method.

Respectfully disagree, as a weaker player using both an AB engine and NN engine, through trial and error, will learn over time which engine to trust in which type of positions. In the US, we call this "OJT", or "On the Job Training". Of course, it will take some work on their part.

During a competition it is important to make less fault.
If you use (powerful) AB and NN engines it is higher the possibility making faults when you use one of them only.
Naturally if it is important to you the participation or the learning than the result the number of faults is unimportant.

[Zenmastur]

It's called Eman XXX. I have no idea about how it works, as apparently it's some private thing owned by a few individuals. But Leela and Stockfish (Eman is strongest derivative) are run in parallel and the system does something to pick a move.

My guess is that there's some positions where Leela is worse than Stockfish and others where Stockfish is worse than Leela, and the system is picking moves from the worse entity, which proves fatal, or something.

So the hybrid performs worse than Stockfish or Leela alone.

That seems a bit odd. I would think it would be easy to figure out when Leela is better and when SF is clearly better. When one side is ahead SF can find good plans easily. When the position is roughly equal SF sucks at analysis. It has trouble finding a plan unless it's later in the game. In the opening stock fish doesn't necessarily suck if it understands the position but when it doesn't understand the position it's play tends to suffer due to lack of a good plan. Leela's play seems to suffer when precise tactics are needed.

I know it's the best hybrid because these people want to win very badly and are using the best that they can, if a better hybrid existed they'd be using that instead. So it turns out we're at a point where the computer plays stronger with the CPU sitting idle than with the strongest A/B engine chunking out moves and scores that just weaken Leela, and Leela is improving faster than hybrid efforts, so unless Leela hits a brick wall at some point (flat lines on her improvements), gambling for hybrid looks very risky.

Right, looks like hybrid is still on its infancy and that a good approach would need to do something different at the learning phase of the NN.

There are a huge number of ways that NN's could be used in an AB engine to improve it's play. I can even imagine using small NN to find new rules that then can be coded normally without putting NN in the engine proper. Mostly for simple things that most engine programmers are just guessing at now. I suspect that LMR and pruning would be much better ( meaning the could be used more aggressively) if controlled by a relatively simple NN.

Yeah, I can't help but think A/B engines are wasting so many resources and that a better multicore strategy must be devised. When I take a look at people with 40 cores playing 10 ply deeper than me I just think I should be losing the game badly, so drawing with ease just means A/B engines are using the extra resources improperly.

The following table was taken from the Alpha Zero pre-print:

Program ..Win ...Draw ..Loss
Stockfish 57.1% 42.9% 0.0 %
Elmo...... 98.7% 01.0% 0.3%
Table S5: Performance comparison of Stockfish and Elmo, when using full time controls of 3h
per game, compared to time controls of 1s per move.


I'm not sure of the exact meaning of the caption but the score represents only a 50 ELO difference for a lot of extra searching. But under what conditions (i.e. selfplay?,opponent play? cache size, NPS, # of cores etc) remains a mystery. But it seems to be echoing your sentiment that extra searching does little for AB engines under some circumstances.

Currently self-learning books have been tried for Eman (with its experience file) and SugarNN (where people start without book and let the thing improve and learn by itself) and for the former Khalid (Eman's author) abandoned it in favor of normal books (the engine uses experience file only after leaving book) and SugarNN is playing weaker than Eman, but I guess it'll improve as time goes by...

There is a “Proper” way to create a book for an engine based on it's true performance as opposed to it's expected win percentage based on it's scoring function. When it's had no experience in a particular line it can fall back on it's expected win percentage based on it's scoring function. But, in order for this to work effectively the program has to be able to choose it's own opening play. Since building the book up to a point that it's useful is a time consuming endeavor one must allow the program to choose it's own openings even during testing and or training. There are several reasons for this but the single largest factor is that the huge amount of time that is given to testing will “hide” the cost of developing the book. Both SF and Leela could implement this type of book if they chose to.

I think the break point is when you analyze the games after and you can't find any mistakes using an AB engine. When this starts happening on a regular basis then I would be worried.

It's already happening on a regular basis with A/B engine Vs. A/B engine, regardless of hardware (unlike 5 years ago, where a hardware like mine was suicide, and this was against top hardware of that time, which is 5 year old hardware by now). It's still happening rarely with A/B Vs. Leela. Looks like Leela's success is arriving at positions where A/B engines are much more likely to blunder, regardless of hardware (A/B engine being "outevaluated" so it doesn't matter what depth they reach), but where Leela is also likely to blunder (again, regardless of hardware, so if the position requires high tactical capability, the strongest piled GPUs will lose against A/B in hardware slower than mine.)

Well, I haven't seen a game where one side won and I couldn't find one or more moves where the losing side made errors. But if you have one that qualifies I'd like to take a look at it.

It does seem to be the case that Leela can find positions that the AB algorithm seem to vassilate between two or more moves in it's iterations with at least one of the moves being a mistake or blunder. This does lead to blunders, wWhich simply shows the weakness of SF evaluation function, too much pruning, and ill advised LMR. But all of these failings could be improved upon.

What hardware do you use? NPS?

Diminishing returns and self-refutation (where after enough analysis you find the best move and must play it, because more analysis leads to refuting it - even if the opponent wouldn't - and second best move "catching up" so you no longer know what move is best) are a huge problem in correspondence games, so that instead of playing your games at a little bit higher level (50/10 time control is long enough that you have enough time to reach the breaking point) you're better off just playing more games.

Upgrading hardware to be able to play 80 games instead of 50 doesn't make sense to me, because there's no benefit in playing more games. The branching factor of chess just doesn't make upgrading cost-effective after you reach a certain point, and I believe Uri is already beyond this point.

I generally like very deep analysis in my games. My problem is that the is so much analysis that it's sometimes hard to find the best move after all the analysis is done. The only game I lost in CC was one in which I transposed moves twice in the same game and made another that was not in any of my analysis. It took three mistakes in one game to force a loss. The last mistake was a transposition mistake I made after I had spent a week on it and found a line that would have equalized the position. Boy was I mad at myself after that move!

Regards,

Zenmastur

[Ovyron]

That seems a bit odd. I would think it would be easy to figure out when Leela is better and when SF is clearly better. When one side is ahead SF can find good plans easily.

Are you a programmer? Perhaps you could program something that is better than current hybrid efforts. Like Aiquiry but running A/B and NN in parallel. Somehow a current problem seems to be that both need to be of about the same level in strength, otherwise, if Leela is on a weak GPU when its move is played it's a worse move than what Stockfish alone would have played, so that the entire game would have been played better with Stockfish alone. An hybrid system is needed that manages to play as good as one of the entities alone, or better, no matter how bad are the suggestions of the other entity most of the time.

There are a huge number of ways that NN's could be used in an AB engine to improve it's play. I can even imagine using small NN to find new rules that then can be coded normally without putting NN in the engine proper. Mostly for simple things that most engine programmers are just guessing at now. I suspect that LMR and pruning would be much better ( meaning the could be used more aggressively) if controlled by a relatively simple NN.

It seems at the current time most people haven't moved to Leela because they already have fast CPU and because the improvement from going from CPU to GPU is not worth the investment (as Leela is better than SF, but not by much.) So if the Leela people can run is subpar, a hybrid effort that makes it play better is not useful to them.

What we need (at least until Leela's edge is significant) is a hybrid implementation that uses Leela (on whatever weak GPU the user already has) to improve Stockfish's play (in the fast CPU people already got for computer chess.) But those haven't focused on improving A/B's engine source code yet.

But it seems to be echoing your sentiment that extra searching does little for AB engines under some circumstances.

I'd go as far as saying "under most circumstances." It's unbelievable that I'm performing only 30 ELO worse than people with fast Intel i7s. I think this is another problem hidden by generic book testing, where sub-optimal positions are reached and in those the extra resources are used effectively, but when the weak side plays into optimal positions the extra resources don't make any difference.

There is a “Proper” way to create a book for an engine based on it's true performance as opposed to it's expected win percentage based on it's scoring function. When it's had no experience in a particular line it can fall back on it's expected win percentage based on it's scoring function. But, in order for this to work effectively the program has to be able to choose it's own opening play.

Heh, recently I was commenting about how someone built a Cluster with that approach in mind, and the Cluster eventually arrived at "1.e4 e5 2.Qf3!?" with its learning and reaching won positions against its opponents, and people were commenting about how sinful that was and about how that variation was bad no matter what, by chess principles. I wonder if opening theory of the future will look absolutely alien compared to what we have today, just like it happened in Go.

Well, I haven't seen a game where one side won and I couldn't find one or more moves where the losing side made errors. But if you have one that qualifies I'd like to take a look at it.

I didn't say the losing side wasn't making errors, I said A/B engines don't see those errors in analysis. What happens is the A/B engine thinks its side has the advantage, and thinks the best play for the opponent is to play for a draw, and it can't refute the draw, so it shows 0.00 scores in analysis. But Leela knows she has the advantage, and then the A/B engine realizes this, it's too late.

I believe here's such a game where Leela defeats a Stockfish derivative running on 12 cores, where I believe A/B engines would show a move as a "mistake" even though the mistake was played several moves earlier:

[pgn][White "Leela"] [Black "CFish 12cores"] [Result "1-0"] 1. Nf3 d5 2. g3 c5 3. Bg2 Nf6 4. O-O e6 5. d3 Nc6 6. e4 Be7 7. Bf4 O-O 8. Ne5 Nxe5 9. Bxe5 Nd7 10. Bf4 e5 11. Bc1 d4 12. Nd2 Nb8 13. f4 Nc6 14. f5 Bd7 15. Nf3 b5 16. g4 f6 17. h4 c4 18. Kh1 Be8 19. Bh3 c3 20. Rg1 cxb2 21. Bxb2 Bb4 22. Bc1 Bc3 23. Rb1 a6 24. g5 Bh5 25. Qe2 Kh8 26. a3 Ra7 27. gxf6 Qxf6 28. Bg4 Bf7 29. Bg5 Qd6 30. Nh2 Ba5 31. Bh5 Bd8 32. Bxf7 Raxf7 33. Nf3 Bf6 34. h5 Ra7 35. Rg3 Bxg5 36. Nxg5 Qh6 37. Qh2 Nd8 38. Rbg1 Rg8 39. Rg4 Nf7 40. Ne6 Nd8 41. Nxd8 Rxd8 42. Qxe5 Qxh5+ 43. Qh2 Qxh2+ 44. Kxh2 g6 45. fxg6 h6 46. Kg3 Kg7 47. e5 Re7 48. Re1 a5 49. Rf4 Rd5 50. Rf5 Re6 51. Kf3 Rxg6 52. Ke4 Rd7 53. Rff1 Kh7 54. Rg1 Rc6 55. Rg2 b4 56. a4 Re6 57. Kf5 Rde7 58. Rg4 h5 59. Rxd4 Rg6 60. Rh4 Rf7+ 61. Ke4 Rg5 62. Kd5 Rf6 63. Re2 Kh6 64. d4 Rf8 65. Rhe4 Kh7 66. Kd6 Rd8+ 67. Kc7 Rgg8 68. e6 Rc8+ 69. Kd6 Rcd8+ 70. Kc5 Rc8+ 71. Kd5 Rcd8+ 72. Kc4 Rc8+ 73. Kb3 Rg5 74. e7 Re8 75. Kc4 Rc8+ 76. Kd3 Re8 77. Kc4 Rc8+ 78. Kd3 b3 79. cxb3 Re8 80. Kc4 Rc8+ 81. Kd3 Re8 82. R2e3 Kg6 83. Kc4 Rc8+ 84. Kd3 Re8 85. Rf3 Kg7 86. Kc4 Rc8+ 87. Kd3 Re8 88. Kc3 Rc8+ 89. Kd2 Re8 90. Kd3 Kg8 91. Kc4 Rc8+ 92. Kd3 Re8 93. Rfe3 Kf7 94. Rf4+ Kg8 95. Kc4 Rc8+ 96. Kd3 Re8 97. Re6 Rd5 98. Kc4 Rd7 99. Re5 Rc7+ 100. Kd3 Rcxe7 101. Rxh5 Re3+ 102. Kc4 Kg7 103. Rg5+ Kh6 104. Rxa5 Re1 105. Rb5 Rc1+ 106. Kb4 Re3 107. Rf8 Kg7 108. Rd8 Rb1 109. Kc4 Rc1+ 110. Kd5 Rf3 111. Ke4 Rf2 112. Rb7+ Kg6 113. Rg8+ Kh6 114. Rb6+ Kh5 115. a5 Re1+ 116. Kd5 Rf5+ 117. Kc4 Rc1+ 118. Kb4 Rf3 119. Rb7 Kh6 120. Rh8+ Kg6 121. Rb6+ Kg5 122. Rb5+ Kg6 123. Rg8+ Kf7 124. Rgb8 Rd3 125. Rd5 Ke6 126. Rc5 Rxd4+ 127. Kb5 Rb1 128. Rc3 Rd5+ 129. Ka6 Rdd1 130. Rb6+ Ke5 131. b4 Ra1 132. Rc5+ Kf4 133. Re6 Rd8 134. Kb7 Kf3 135. Rf6+ Ke3 136. a6 Rd4 137. b5 Rda4 138. Rf7 R4a2 139. Rf8 Kd4 140. Rcf5 Re2 141. R8f7 Re8 142. Rc7 Ke4 143. Rh5 Ke3 144. a7 Rf8 145. b6 Kf4 146. Rc8 Rf7+ 147. Rc7 Rf8 148. Rhh7 Ke3 149. Rhd7 Ke2 150. Rc8 Rf6 151. a8=Q Rxa8 152. Rxa8 Re6 153. Kc7 Re3 154. b7 Rc3+ 155. Kd8 Rb3 156. b8=N Rc3 157. Nc6 Rxc6 158. Re7+ Kf3 159. Kd7 Rg6 160. Rf8+ Kg2 161. Ref7 Kg3 162. Rh8 Rg4 163. Ke8 Re4+ 164. Kf8 Rf4 165. Rxf4 Kxf4 166. Ke7 Ke5 167. Rh4 Kd5 168. Kf6 Kd6 169. Rh5 Kd7 170. Rd5+ Kc6 171. Ke6 Kc7 172. Rc5+ Kb7 173. Kd7 Kb6 174. Rh5 Kb7 175. Rh6 Kb8 176. Kc6 Ka7 177. Rh8 Ka6 178. Ra8# 1-0[/pgn]

The opposite also happens, here's a game where Leela running on the fastest GPUs I know of lost against a Stockfish derivative running on hardware slower than mine:

[pgn][White "Leela"] [Black "Eman on slow hardware"] [Result "0-1"] 1. Nf3 d5 2. g3 Nf6 3. Bg2 c6 4. c4 dxc4 5. O-O Nbd7 6. Qc2 Nb6 7. Na3 Be6 8. Ne5 h5 9. Naxc4 Nxc4 10. Nxc4 h4 11. Ne5 hxg3 12. hxg3 Qd4 13. Qc3 Qd6 14. d4 Rd8 15. Qa5 Qxd4 16. Be3 Qd6 17. Bf4 Rh5 18. Bf3 Bg4 19. Bg2 g6 20. Qc3 Qe6 21. Nxg4 Nxg4 22. Rfd1 Rxd1+ 23. Rxd1 f5 24. Bf3 g5 25. Bc7 Qh6 26. Rd8+ Kf7 27. Qc4+ e6 28. Bxg4 Rh1+ 29. Kg2 fxg4 30. Qxg4 Rc1 31. Qf3+ Ke7 32. Qa3+ c5 33. Kf3 g4+ 34. Kxg4 Qg6+ 35. Kf3 Bh6 36. e3 Qh5+ 37. g4 Qh3+ 38. Bg3 Qh1+ 39. Ke2 Qf1+ 40. Kf3 Kxd8 41. Qa5+ Ke8 42. Qa4+ b5 43. Qb3 Qh1+ 44. Ke2 Kf7 45. Qd3 Re1+ 46. Kd2 Rd1+ 47. Kc2 Rxd3 48. b3 Qd1+ 49. Kb2 Rd2+ 50. Ka3 Qc1# 0-1 [/pgn]

What hardware do you use? NPS?

Q8200 2.33 GHz (I bought this one specifically because it was easy to overclock and it's a quiet CPU, but I never did it because for all my needs it was fast enough.)
128MB RAM (never found any advantage whatsoever from giving more than this to engines)
3CPU (the GUI is buggy and requires an entire core dedicated to it to remain stable...)
Around 2000 kn/s in average.

I generally like very deep analysis in my games. My problem is that the is so much analysis that it's sometimes hard to find the best move after all the analysis is done. The only game I lost in CC was one in which I transposed moves twice in the same game and made another that was not in any of my analysis. It took three mistakes in one game to force a loss. The last mistake was a transposition mistake I made after I had spent a week on it and found a line that would have equalized the position. Boy was I mad at myself after that move!

Have you tried playing at the LSS? (chess-server) I have found that people tend to cause more trouble there when compared to similarly rated opponents from other sites, and if you already have an ICCF rating, they allow you to start there so you skip the three years it'd require you to beat weaklings to reach it.

[zullil]

I didn't say the losing side wasn't making errors, I said A/B engines don't see those errors in analysis. What happens is the A/B engine thinks its side has the advantage, and thinks the best play for the opponent is to play for a draw, and it can't refute the draw, so it shows 0.00 scores in analysis. But Leela knows she has the advantage, and then the A/B engine realizes this, it's too late.

I believe here's such a game where Leela defeats a Stockfish derivative running on 12 cores, where I believe A/B engines would show a move as a "mistake" even though the mistake was played several moves earlier:

If the A/B engine realizes it "made a mistake" with an earlier move, then, given enough time, it would have switched away from that move during the earlier search.

All you're describing here is that A/B search with its various prunings and reductions sometimes fails due to insufficient depth/time.

Where, in the Cfish game, is a position of the sort you are discussing? Where Cfish shows 0.00 and then later realizes it is losing. I'm sure with a longer search that 0.00 would vanish.

[Ovyron]

How long? Depth 80? 2 weeks of analysis time? Who cares about that if the engine is never going to be put to analyze that long, and this position isn't ever going to be seen in a game again?

Yet Leela will show the mistake in under a minute in a decent GPU, so this isn't about search at all, but about "outevaluating" the opponent. With the correct evaluation terms implemented, Stockfish would also show the mistake under a minute.

[zullil]

How long? Depth 80? 2 weeks of analysis time? Who cares about that if the engine is never going to be put to analyze that long, and this position isn't ever going to be seen in a game again?

Yet Leela will show the mistake in under a minute in a decent GPU, so this isn't about search at all, but about "outevaluating" the opponent. With the correct evaluation terms implemented, Stockfish would also show the mistake under a minute.

Like any evaluation cobbled together by humans, Stockfish's evaluation is far, far from perfect. That Stockfish plays as well as it does suggests that a state-of-the-art search can compensate for a lot of shortcomings in evaluation.

Still, can you point to a specific position in the Cfish game that you feel best illustrates your claim? I've been looking a bit at

[d]r2qbrk1/p3b1pp/2n2p2/1p2pP2/3pP1PP/3P1N1B/PBP5/R2Q2RK b - - 0 21

Here Cfish played 21...Bb4 and things went quickly downhill from there.

By the way, do you know the time control for this game?

[Ovyron]

Here Cfish played 21...Bb4 and things went quickly downhill from there.

Are you claiming that that's THE mistake? That black played perfectly up to that point?

By the way, do you know the time control for this game?

12 +2

[corres]

...
There is a “Proper” way to create a book for an engine based on it's true performance as opposed to it's expected win percentage based on it's scoring function. When it's had no experience in a particular line it can fall back on it's expected win percentage based on it's scoring function. But, in order for this to work effectively the program has to be able to choose it's own opening play. Since building the book up to a point that it's useful is a time consuming endeavor one must allow the program to choose it's own openings even during testing and or training. There are several reasons for this but the single largest factor is that the huge amount of time that is given to testing will “hide” the cost of developing the book. Both SF and Leela could implement this type of book if they chose to.
...

Some note:
Only the NN engines (like Leela) has its "own openings".
AB engines have only tendency to play lines from start position.
If you want to create a good book for an AB engine you need huge number of games between that engine and many other engines to know the weakness of that AB engine. The book should be able to help the engine to divert lines in what the engine is weak and to keep the engine on the winning lines.

[zullil]

Here Cfish played 21...Bb4 and things went quickly downhill from there.

Are you claiming that that's THE mistake? That black played perfectly up to that point?

By the way, do you know the time control for this game?

12 +2

I simply said I was looking at that particular position. If I have some time, I might look some more at the game you posted. Since chess is "obviously" a draw with perfect play and yet Cfish lost the game, there is indeed at least one move by Cfish which changed the true evaluation of game from DRAW to LOSS. If you already believe you know the first such move, maybe you could save me some time by sharing your insight.

[Zenmastur]

Here Cfish played 21...Bb4 and things went quickly downhill from there.

Are you claiming that that's THE mistake? That black played perfectly up to that point?

By the way, do you know the time control for this game?

12 +2

I simply said I was looking at that particular position. If I have some time, I might look some more at the game you posted. Since chess is "obviously" a draw with perfect play and yet Cfish lost the game, there is indeed at least one move by Cfish which changed the true evaluation of game from DRAW to LOSS. If you already believe you know the first such move, maybe you could save me some time by sharing your insight.

It looks like 14... Bd7 wasn't the best move.

Code: Select all

(  [Stockfish 010719 64 BMI2] 50:+0.00 14...Bg5 15.h4 Be3+ 16.Kh2 f6 17.Nf3 Bxc1 18.Rxc1 Qb6 19.Rb1 Kh8 20.Qe2 Bd7 21.Bh3 Qc7 22.Rg1 Ne7 23.Nd2 b6 24.Rbf1 h6 25.Bg4 Rg8 26.Bh5 Raf8 27.b3 Kh7 28.a4 Qc6 29.Nc4 Bc8 30.Qf3 Qc7 31.Nd2 Bd7 32.Ra1 Qb7 33.Nc4 Qc6 34.Bg4 Qc7 35.Nd2 Rc8 36.Bh5 Rcf8 37.Qd1 Qc6 38.g4 g6 39.Bxg6+ Nxg6 40.fxg6+ Rxg6 )
(  [Stockfish 010719 64 BMI2] 50:+0.00 14...a5 15.Nf3 a4 16.g4 f6 17.Rf2 g5 18.h4 h6 19.Bf1 Kf7 20.Rh2 Bd7 21.Bd2 Qc7 22.Qc1 Ke8 23.Rb1 Kd8 24.Be1 Kc8 25.a3 b6 26.h5 Bd6 27.c4 Na5 28.Nd2 Nb3 29.Qd1 Na5 30.Qc2 Bc6 31.Qc1 Nb3 32.Qc2 )
(  [Stockfish 010719 64 BMI2] 50:+0.00 14...b5 15.a4 bxa4 16.Rxa4 a5 17.h4 Ba6 18.Ra1 Qc7 19.Kh2 Rfb8 20.Nc4 Qd8 21.Rf3 Qc7 )

It's been so long since I posted on this forum I forgot how to post a board!

Regards,

Zenmastur