buying a new computer

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
corres
Posts: 1779
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: buying a new computer

Post by corres » Thu Jul 11, 2019 8:18 am

MikeB wrote:
Wed Jul 10, 2019 5:54 pm
corres wrote:
Wed Jul 10, 2019 2:08 pm
In general if one uses two systems like an AB engine and an NN engine for making analysis that one should be a good chess player because only that one should choose the right move if there is a difference in the value of the analyzed position between the two engines.
So for a weaker chess player it is better to use only one - AB or NN - system than two.
Maybe a cross analysis can help in this situation too but it is a very time consuming method.
Respectfully disagree, as a weaker player using both an AB engine and NN engine, through trial and error, will learn over time which engine to trust in which type of positions. In the US, we call this "OJT", or "On the Job Training". Of course, it will take some work on their part.
During a competition it is important to make less fault.
If you use (powerful) AB and NN engines it is higher the possibility making faults when you use one of them only.
Naturally if it is important to you the participation or the learning than the result the number of faults is unimportant.

Zenmastur
Posts: 515
Joined: Sat May 31, 2014 6:28 am

Re: buying a new computer

Post by Zenmastur » Thu Jul 11, 2019 10:21 am

Ovyron wrote:
Thu Jul 11, 2019 1:48 am

It's called Eman XXX. I have no idea about how it works, as apparently it's some private thing owned by a few individuals. But Leela and Stockfish (Eman is strongest derivative) are run in parallel and the system does something to pick a move.

My guess is that there's some positions where Leela is worse than Stockfish and others where Stockfish is worse than Leela, and the system is picking moves from the worse entity, which proves fatal, or something.

So the hybrid performs worse than Stockfish or Leela alone.
That seems a bit odd. I would think it would be easy to figure out when Leela is better and when SF is clearly better. When one side is ahead SF can find good plans easily. When the position is roughly equal SF sucks at analysis. It has trouble finding a plan unless it's later in the game. In the opening stock fish doesn't necessarily suck if it understands the position but when it doesn't understand the position it's play tends to suffer due to lack of a good plan. Leela's play seems to suffer when precise tactics are needed.
I know it's the best hybrid because these people want to win very badly and are using the best that they can, if a better hybrid existed they'd be using that instead. So it turns out we're at a point where the computer plays stronger with the CPU sitting idle than with the strongest A/B engine chunking out moves and scores that just weaken Leela, and Leela is improving faster than hybrid efforts, so unless Leela hits a brick wall at some point (flat lines on her improvements), gambling for hybrid looks very risky.
Right, looks like hybrid is still on its infancy and that a good approach would need to do something different at the learning phase of the NN.
There are a huge number of ways that NN's could be used in an AB engine to improve it's play. I can even imagine using small NN to find new rules that then can be coded normally without putting NN in the engine proper. Mostly for simple things that most engine programmers are just guessing at now. I suspect that LMR and pruning would be much better ( meaning the could be used more aggressively) if controlled by a relatively simple NN.
Yeah, I can't help but think A/B engines are wasting so many resources and that a better multicore strategy must be devised. When I take a look at people with 40 cores playing 10 ply deeper than me I just think I should be losing the game badly, so drawing with ease just means A/B engines are using the extra resources improperly.
The following table was taken from the Alpha Zero pre-print:

Program ..Win ...Draw ..Loss
Stockfish 57.1% 42.9% 0.0 %
Elmo...... 98.7% 01.0% 0.3%
Table S5: Performance comparison of Stockfish and Elmo, when using full time controls of 3h
per game, compared to time controls of 1s per move.


I'm not sure of the exact meaning of the caption but the score represents only a 50 ELO difference for a lot of extra searching. But under what conditions (i.e. selfplay?,opponent play? cache size, NPS, # of cores etc) remains a mystery. But it seems to be echoing your sentiment that extra searching does little for AB engines under some circumstances.
Currently self-learning books have been tried for Eman (with its experience file) and SugarNN (where people start without book and let the thing improve and learn by itself) and for the former Khalid (Eman's author) abandoned it in favor of normal books (the engine uses experience file only after leaving book) and SugarNN is playing weaker than Eman, but I guess it'll improve as time goes by...
There is a “Proper” way to create a book for an engine based on it's true performance as opposed to it's expected win percentage based on it's scoring function. When it's had no experience in a particular line it can fall back on it's expected win percentage based on it's scoring function. But, in order for this to work effectively the program has to be able to choose it's own opening play. Since building the book up to a point that it's useful is a time consuming endeavor one must allow the program to choose it's own openings even during testing and or training. There are several reasons for this but the single largest factor is that the huge amount of time that is given to testing will “hide” the cost of developing the book. Both SF and Leela could implement this type of book if they chose to.
Zenmastur wrote:I think the break point is when you analyze the games after and you can't find any mistakes using an AB engine. When this starts happening on a regular basis then I would be worried.
It's already happening on a regular basis with A/B engine Vs. A/B engine, regardless of hardware (unlike 5 years ago, where a hardware like mine was suicide, and this was against top hardware of that time, which is 5 year old hardware by now). It's still happening rarely with A/B Vs. Leela. Looks like Leela's success is arriving at positions where A/B engines are much more likely to blunder, regardless of hardware (A/B engine being "outevaluated" so it doesn't matter what depth they reach), but where Leela is also likely to blunder (again, regardless of hardware, so if the position requires high tactical capability, the strongest piled GPUs will lose against A/B in hardware slower than mine.)
Well, I haven't seen a game where one side won and I couldn't find one or more moves where the losing side made errors. But if you have one that qualifies I'd like to take a look at it.

It does seem to be the case that Leela can find positions that the AB algorithm seem to vassilate between two or more moves in it's iterations with at least one of the moves being a mistake or blunder. This does lead to blunders, wWhich simply shows the weakness of SF evaluation function, too much pruning, and ill advised LMR. But all of these failings could be improved upon.

What hardware do you use? NPS?
Diminishing returns and self-refutation (where after enough analysis you find the best move and must play it, because more analysis leads to refuting it - even if the opponent wouldn't - and second best move "catching up" so you no longer know what move is best) are a huge problem in correspondence games, so that instead of playing your games at a little bit higher level (50/10 time control is long enough that you have enough time to reach the breaking point) you're better off just playing more games.

Upgrading hardware to be able to play 80 games instead of 50 doesn't make sense to me, because there's no benefit in playing more games. The branching factor of chess just doesn't make upgrading cost-effective after you reach a certain point, and I believe Uri is already beyond this point.
I generally like very deep analysis in my games. My problem is that the is so much analysis that it's sometimes hard to find the best move after all the analysis is done. The only game I lost in CC was one in which I transposed moves twice in the same game and made another that was not in any of my analysis. It took three mistakes in one game to force a loss. The last mistake was a transposition mistake I made after I had spent a week on it and found a line that would have equalized the position. Boy was I mad at myself after that move!

Regards,

Zenmastur
Only 2 defining forces have ever offered to die for you.....Jesus Christ and the American Soldier. One died for your soul, the other for your freedom.

User avatar
Ovyron
Posts: 2828
Joined: Tue Jul 03, 2007 2:30 am

Re: buying a new computer

Post by Ovyron » Fri Jul 12, 2019 12:05 am

Zenmastur wrote:
Thu Jul 11, 2019 10:21 am
That seems a bit odd. I would think it would be easy to figure out when Leela is better and when SF is clearly better. When one side is ahead SF can find good plans easily.
Are you a programmer? Perhaps you could program something that is better than current hybrid efforts. Like Aiquiry but running A/B and NN in parallel. Somehow a current problem seems to be that both need to be of about the same level in strength, otherwise, if Leela is on a weak GPU when its move is played it's a worse move than what Stockfish alone would have played, so that the entire game would have been played better with Stockfish alone. An hybrid system is needed that manages to play as good as one of the entities alone, or better, no matter how bad are the suggestions of the other entity most of the time.
Zenmastur wrote:
Thu Jul 11, 2019 10:21 am
There are a huge number of ways that NN's could be used in an AB engine to improve it's play. I can even imagine using small NN to find new rules that then can be coded normally without putting NN in the engine proper. Mostly for simple things that most engine programmers are just guessing at now. I suspect that LMR and pruning would be much better ( meaning the could be used more aggressively) if controlled by a relatively simple NN.
It seems at the current time most people haven't moved to Leela because they already have fast CPU and because the improvement from going from CPU to GPU is not worth the investment (as Leela is better than SF, but not by much.) So if the Leela people can run is subpar, a hybrid effort that makes it play better is not useful to them.

What we need (at least until Leela's edge is significant) is a hybrid implementation that uses Leela (on whatever weak GPU the user already has) to improve Stockfish's play (in the fast CPU people already got for computer chess.) But those haven't focused on improving A/B's engine source code yet.
But it seems to be echoing your sentiment that extra searching does little for AB engines under some circumstances.
I'd go as far as saying "under most circumstances." It's unbelievable that I'm performing only 30 ELO worse than people with fast Intel i7s. I think this is another problem hidden by generic book testing, where sub-optimal positions are reached and in those the extra resources are used effectively, but when the weak side plays into optimal positions the extra resources don't make any difference.
There is a “Proper” way to create a book for an engine based on it's true performance as opposed to it's expected win percentage based on it's scoring function. When it's had no experience in a particular line it can fall back on it's expected win percentage based on it's scoring function. But, in order for this to work effectively the program has to be able to choose it's own opening play.
Heh, recently I was commenting about how someone built a Cluster with that approach in mind, and the Cluster eventually arrived at "1.e4 e5 2.Qf3!?" with its learning and reaching won positions against its opponents, and people were commenting about how sinful that was and about how that variation was bad no matter what, by chess principles. I wonder if opening theory of the future will look absolutely alien compared to what we have today, just like it happened in Go.
Well, I haven't seen a game where one side won and I couldn't find one or more moves where the losing side made errors. But if you have one that qualifies I'd like to take a look at it.
I didn't say the losing side wasn't making errors, I said A/B engines don't see those errors in analysis. What happens is the A/B engine thinks its side has the advantage, and thinks the best play for the opponent is to play for a draw, and it can't refute the draw, so it shows 0.00 scores in analysis. But Leela knows she has the advantage, and then the A/B engine realizes this, it's too late.

I believe here's such a game where Leela defeats a Stockfish derivative running on 12 cores, where I believe A/B engines would show a move as a "mistake" even though the mistake was played several moves earlier:



The opposite also happens, here's a game where Leela running on the fastest GPUs I know of lost against a Stockfish derivative running on hardware slower than mine:


What hardware do you use? NPS?
Q8200 2.33 GHz (I bought this one specifically because it was easy to overclock and it's a quiet CPU, but I never did it because for all my needs it was fast enough.)
128MB RAM (never found any advantage whatsoever from giving more than this to engines)
3CPU (the GUI is buggy and requires an entire core dedicated to it to remain stable...)
Around 2000 kn/s in average.
I generally like very deep analysis in my games. My problem is that the is so much analysis that it's sometimes hard to find the best move after all the analysis is done. The only game I lost in CC was one in which I transposed moves twice in the same game and made another that was not in any of my analysis. It took three mistakes in one game to force a loss. The last mistake was a transposition mistake I made after I had spent a week on it and found a line that would have equalized the position. Boy was I mad at myself after that move!
Have you tried playing at the LSS? (chess-server) I have found that people tend to cause more trouble there when compared to similarly rated opponents from other sites, and if you already have an ICCF rating, they allow you to start there so you skip the three years it'd require you to beat weaklings to reach it.
Great spirits have always encountered violent opposition from mediocre minds.

zullil
Posts: 5684
Joined: Mon Jan 08, 2007 11:31 pm
Location: PA USA
Full name: Louis Zulli

Re: buying a new computer

Post by zullil » Fri Jul 12, 2019 12:30 am

Ovyron wrote:
Fri Jul 12, 2019 12:05 am

I didn't say the losing side wasn't making errors, I said A/B engines don't see those errors in analysis. What happens is the A/B engine thinks its side has the advantage, and thinks the best play for the opponent is to play for a draw, and it can't refute the draw, so it shows 0.00 scores in analysis. But Leela knows she has the advantage, and then the A/B engine realizes this, it's too late.

I believe here's such a game where Leela defeats a Stockfish derivative running on 12 cores, where I believe A/B engines would show a move as a "mistake" even though the mistake was played several moves earlier:
If the A/B engine realizes it "made a mistake" with an earlier move, then, given enough time, it would have switched away from that move during the earlier search.

All you're describing here is that A/B search with its various prunings and reductions sometimes fails due to insufficient depth/time.

Where, in the Cfish game, is a position of the sort you are discussing? Where Cfish shows 0.00 and then later realizes it is losing. I'm sure with a longer search that 0.00 would vanish.

User avatar
Ovyron
Posts: 2828
Joined: Tue Jul 03, 2007 2:30 am

Re: buying a new computer

Post by Ovyron » Fri Jul 12, 2019 12:52 am

How long? Depth 80? 2 weeks of analysis time? Who cares about that if the engine is never going to be put to analyze that long, and this position isn't ever going to be seen in a game again?

Yet Leela will show the mistake in under a minute in a decent GPU, so this isn't about search at all, but about "outevaluating" the opponent. With the correct evaluation terms implemented, Stockfish would also show the mistake under a minute.
Great spirits have always encountered violent opposition from mediocre minds.

zullil
Posts: 5684
Joined: Mon Jan 08, 2007 11:31 pm
Location: PA USA
Full name: Louis Zulli

Re: buying a new computer

Post by zullil » Fri Jul 12, 2019 2:31 am

Ovyron wrote:
Fri Jul 12, 2019 12:52 am
How long? Depth 80? 2 weeks of analysis time? Who cares about that if the engine is never going to be put to analyze that long, and this position isn't ever going to be seen in a game again?

Yet Leela will show the mistake in under a minute in a decent GPU, so this isn't about search at all, but about "outevaluating" the opponent. With the correct evaluation terms implemented, Stockfish would also show the mistake under a minute.
Like any evaluation cobbled together by humans, Stockfish's evaluation is far, far from perfect. That Stockfish plays as well as it does suggests that a state-of-the-art search can compensate for a lot of shortcomings in evaluation.

Still, can you point to a specific position in the Cfish game that you feel best illustrates your claim? I've been looking a bit at



Here Cfish played 21...Bb4 and things went quickly downhill from there.

By the way, do you know the time control for this game?

User avatar
Ovyron
Posts: 2828
Joined: Tue Jul 03, 2007 2:30 am

Re: buying a new computer

Post by Ovyron » Fri Jul 12, 2019 2:57 am

zullil wrote:
Fri Jul 12, 2019 2:31 am
Here Cfish played 21...Bb4 and things went quickly downhill from there.
Are you claiming that that's THE mistake? That black played perfectly up to that point?
zullil wrote:
Fri Jul 12, 2019 2:31 am
By the way, do you know the time control for this game?
12 +2
Great spirits have always encountered violent opposition from mediocre minds.

corres
Posts: 1779
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: buying a new computer

Post by corres » Fri Jul 12, 2019 7:58 am

Zenmastur wrote:
Thu Jul 11, 2019 10:21 am
...
There is a “Proper” way to create a book for an engine based on it's true performance as opposed to it's expected win percentage based on it's scoring function. When it's had no experience in a particular line it can fall back on it's expected win percentage based on it's scoring function. But, in order for this to work effectively the program has to be able to choose it's own opening play. Since building the book up to a point that it's useful is a time consuming endeavor one must allow the program to choose it's own openings even during testing and or training. There are several reasons for this but the single largest factor is that the huge amount of time that is given to testing will “hide” the cost of developing the book. Both SF and Leela could implement this type of book if they chose to.
...
Some note:
Only the NN engines (like Leela) has its "own openings".
AB engines have only tendency to play lines from start position.
If you want to create a good book for an AB engine you need huge number of games between that engine and many other engines to know the weakness of that AB engine. The book should be able to help the engine to divert lines in what the engine is weak and to keep the engine on the winning lines.

zullil
Posts: 5684
Joined: Mon Jan 08, 2007 11:31 pm
Location: PA USA
Full name: Louis Zulli

Re: buying a new computer

Post by zullil » Fri Jul 12, 2019 10:31 am

Ovyron wrote:
Fri Jul 12, 2019 2:57 am
zullil wrote:
Fri Jul 12, 2019 2:31 am
Here Cfish played 21...Bb4 and things went quickly downhill from there.
Are you claiming that that's THE mistake? That black played perfectly up to that point?
zullil wrote:
Fri Jul 12, 2019 2:31 am
By the way, do you know the time control for this game?
12 +2
I simply said I was looking at that particular position. If I have some time, I might look some more at the game you posted. Since chess is "obviously" a draw with perfect play and yet Cfish lost the game, there is indeed at least one move by Cfish which changed the true evaluation of game from DRAW to LOSS. If you already believe you know the first such move, maybe you could save me some time by sharing your insight. :D

Zenmastur
Posts: 515
Joined: Sat May 31, 2014 6:28 am

Re: buying a new computer

Post by Zenmastur » Fri Jul 12, 2019 8:38 pm

zullil wrote:
Fri Jul 12, 2019 10:31 am
Ovyron wrote:
Fri Jul 12, 2019 2:57 am
zullil wrote:
Fri Jul 12, 2019 2:31 am
Here Cfish played 21...Bb4 and things went quickly downhill from there.
Are you claiming that that's THE mistake? That black played perfectly up to that point?
zullil wrote:
Fri Jul 12, 2019 2:31 am
By the way, do you know the time control for this game?
12 +2
I simply said I was looking at that particular position. If I have some time, I might look some more at the game you posted. Since chess is "obviously" a draw with perfect play and yet Cfish lost the game, there is indeed at least one move by Cfish which changed the true evaluation of game from DRAW to LOSS. If you already believe you know the first such move, maybe you could save me some time by sharing your insight. :D
It looks like 14... Bd7 wasn't the best move.

Code: Select all

(  [Stockfish 010719 64 BMI2] 50:+0.00 14...Bg5 15.h4 Be3+ 16.Kh2 f6 17.Nf3 Bxc1 18.Rxc1 Qb6 19.Rb1 Kh8 20.Qe2 Bd7 21.Bh3 Qc7 22.Rg1 Ne7 23.Nd2 b6 24.Rbf1 h6 25.Bg4 Rg8 26.Bh5 Raf8 27.b3 Kh7 28.a4 Qc6 29.Nc4 Bc8 30.Qf3 Qc7 31.Nd2 Bd7 32.Ra1 Qb7 33.Nc4 Qc6 34.Bg4 Qc7 35.Nd2 Rc8 36.Bh5 Rcf8 37.Qd1 Qc6 38.g4 g6 39.Bxg6+ Nxg6 40.fxg6+ Rxg6 )
(  [Stockfish 010719 64 BMI2] 50:+0.00 14...a5 15.Nf3 a4 16.g4 f6 17.Rf2 g5 18.h4 h6 19.Bf1 Kf7 20.Rh2 Bd7 21.Bd2 Qc7 22.Qc1 Ke8 23.Rb1 Kd8 24.Be1 Kc8 25.a3 b6 26.h5 Bd6 27.c4 Na5 28.Nd2 Nb3 29.Qd1 Na5 30.Qc2 Bc6 31.Qc1 Nb3 32.Qc2 )
(  [Stockfish 010719 64 BMI2] 50:+0.00 14...b5 15.a4 bxa4 16.Rxa4 a5 17.h4 Ba6 18.Ra1 Qc7 19.Kh2 Rfb8 20.Nc4 Qd8 21.Rf3 Qc7 )
It's been so long since I posted on this forum I forgot how to post a board! :shock: :oops: :roll:

Regards,

Zenmastur
Only 2 defining forces have ever offered to die for you.....Jesus Christ and the American Soldier. One died for your soul, the other for your freedom.

Post Reply