Something Hikaru Said

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

Jesse Gersenson
Posts: 593
Joined: Sat Aug 20, 2011 9:43 am

Re: Something Hikaru Said

Post by Jesse Gersenson »

bob wrote: I did a quick online search and did not find any results produced after the game solution was announced...
bob wrote: not AFTER they finished the final endgame tables that let them play perfectly. He himself had branded it "unbeatable".
My mistake, I'd overlooked the stipulation of 'after game solution was announced'.

Checkers was 'solved' in 2007(?). Tinsley was already 65 when he beat Chinook in their first match (1992) and was 68 when he played it again, drawing 6 games before retreating to die.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Something Hikaru Said

Post by Laskos »

mhull wrote: Ah, of course. I should have thought of that. Thank you for that insight. However, would then the problem be the same only inverted? The optimization is done under short time constraint which my be sub-optimal in championship time control and therefore limit-optimization factors may still be in play.

I don't know if Bob has any insight from cluster testing where longer TC optimization is more practical vis-a-vis short TC optimization.
It might happen, although this is not my main source of doubt about my result. The decrease in the ELO gain for doubling at longer TC is due to increasing draw rate, but win over loss ratio remains pretty constant or even increases with time control. It seems simply Komodo is becoming too strong to gain at long TC as much as at short TC, not that it scales badly, and the wish of the developers to have good scaling of patches to long TC is very much satisfied.

My main concern is that the result reflects more the intrinsic limit of modern engines, i.e. if they all continue in this paradigm, the engines wouldn't go much beyond 1,000 more ELO points above today, but the perfect play might still be much higher. But as of now, I would bet on the result to be close to what it wants to predict.

I tested the same thing with a 10 year old engine Glaurung 1.2.1 (hard to find reliable CEGT and CCRL results useful to me older than this, and they are crucial, as I cannot have many games at their long time control), and the predicted ELO of the perfect engine is close to what happened with Komodo. Seems to be a universal quantity, and this gives some more confidence.

Also, in games in one minute, Mephisto Gideon of 1993 plays perfectly about 30% of non-trivial 5-men endgames against tablebases, while Komodo 9.3 about 95%. At least in this case there is a clear shift from pretty much clueless engines of 20+ years ago, and almost perfect engines on 5-men chess of today. It well could be that some regular games (say starting with 30 pieces) of today's top engines which are wins or draws are actually already perfect play.
Gurcan Uckardes
Posts: 196
Joined: Wed Oct 29, 2014 12:42 am

Re: Something Hikaru Said

Post by Gurcan Uckardes »

I suggest the name of the topic changes to "Something Kai said and Bob refused" :D
By the way, it's all interesting...
My blog for Android users: http://chesstroid.blogspot.com
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Something Hikaru Said

Post by bob »

Gurcan Uckardes wrote:I suggest the name of the topic changes to "Something Kai said and Bob refused" :D
By the way, it's all interesting...
Kai didn't start the thread.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Something Hikaru Said

Post by bob »

Uri Blass wrote:
bob wrote:
Uri Blass wrote:
bob wrote:
Uri Blass wrote:
bob wrote:
Uri Blass wrote:
bob wrote:

None of this seems reasonable to me. The percentage of GM errors, for example. A GM makes far FEWER errors against a weak program than against a strong program, but not because he actually makes fewer errors, but because the opponent doesn't notice them and doesn't punish them.
I disagree here.
If the opponent does not play well it is easier not to make mistakes.

I clearly have games against humans when I did no significant mistakes based on computer analysis(no move reduce the evaluation by more than 0.2 pawn).

It is not because I am so strong but because it is easier not to make mistakes when the opponent does not play well.
If the opponent play well I expect myself to do more mistakes.

Uri
"no significant mistakes based on computer analysis" is meaningless when we are talking about PERFECT computer play. ANY mistake will be significant there. This extrapolation about what happens today is meaningless when we talk about a perfect chess opponent.
It is not meaningless because the computer is clearly stronger than me and find many mistakes.

If the computer find that I play more mistakes in games that I play against stronger opponents it means that it is easier to make mistakes against stronger opponents.

I do not have the 32 piece tablebases but my speculation is that there are many games when the winner did not do a mistake that change the theoretical result and also there are draws with no mistakes.
Of course the side who did no mistake could do mistakes in case of playing against a stronger opponent.
It is not easier to make mistakes against stronger opponents. It is simply more likely that they will understand your mistake and how they can exploit it. Your propensity for making mistakes is independent of the opponent, your brain cells don't suddenly change when you play a 2500 player vs a 1500 player.

As far as the "many games with no discernible mistakes" there are only "many games" because there are millions and millions of games that have been played. If your opponent spots the mistake and beats you only 1% of the time, that does NOT mean you played 99% of your games with no mistakes.
It is easier to make mistakes when the opponent help you to do mistakes.
If the opponent play weak moves I simply has good chances not to get positions when I do mistakes.

It is not that my brain change when I play against stronger player but
I simply need to solve harder problems that I cannot solve when
I do not need to solve hard problems against the weak player.

The opening moves I guess that I know to play perfectly because there are many drawing moves and after the opponent made a mistake it is easy not to make a mistake.

As an extreme example suppose that
I play against a player who make random moves.

I guess that part of the games are going to be 1.f3 e5 2.g4 Qh4 mate when I played perfectly because the opponent did not help me to do mistakes.

You can claim that maybe 1...e5 is a mistake(and I am not 100% sure because maybe 1...e5 draw when 1...d5 win) but my guess is that it is not a mistake.

Against non random player that is a very weak chess player something like the following can happen when again I cannot prove a mistake by white with today's software.

1.e4 e5 2.Nf3 Nf6 3.Nxe5 Nxe4 4.Qe2 Nf6 5.Nc6+ that win the queen can happen when after 5.Nc6+ I believe that it is easy to play perfectly(maybe I do not mate in the fastest way that is not important but I always play winning moves).
Your opponent can't help YOU "do mistakes". Not possible. He doesn't get to make moves for you. Both of you will still make mistakes at the same rate as always, but a stronger player will make fewer, and in particular fewer that the weaker player can grasp and punish. But he still makes mistakes just the same.

You are mixing playing skill with this mistake stuff. A player at a given level makes mistakes at a predictable rate. It just takes a stronger player to spot many of them. But just because they are not spotted, doesn't mean they are not there.
The stronger player make moves that lead to position when it is easier for the weaker player to make mistakes.

If you take a game that is solved like checker and 2 human non perfect players when one is significantly stronger but still usually lose against the perfect player then I believe that analysis can show that the winner often does not make mistakes based on computer analysis because the opponent made first a mistake and after the mistake it was an easy win.
I don't agree. We are not using the same definition of "mistake". A player of a given skill will make some predictable number of "outcome-altering" mistakes in a game. That's where his rating comes from in the first place. But his rating is not based only on the number of mistakes, it is based on the number of mistakes he makes that his opponent can take advantage of. The best GM on the planet might never lose a game, but that doesn't mean he doesn't make mistakes at his usual rate, it just means nobody can recognize and take advantage of them. But a perfect player will see every last mistake and win because of them.

As of today, nobody has a clue about how many mistakes a GM makes, because you have to first recognize the mistake, and that's often impossible in chess. Until you find a perfect player.

There was a time where some deep blue moves were labeled as "mistakes" based on GM analysis. Deep analysis showed that they were not mistakes, the humans just couldn't follow what was going on. 7 piece endings are already problematic for human understanding. What happens when there are 32 piece files in this hypothetical discussion?
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Something Hikaru Said

Post by bob »

kbhearn wrote:
bob wrote:And every time I've heard it cited as some sort of justification why a computer can never do "x" (x = {beat an expert, beat a master, beat an IM, beat a GM, beat the world champion, beat the world champion in a match, to the present beat the world champion when giving him some sort of handicap from a pawn to a knight or more}. Each time a ultimate barrier was raised, it was brushed aside. Logic says the same will happen here. That there must be an ultimate handicap beyond which nobody can win is obviously true. But what that ultimate handicap might end up at is today completely unknown, other than via wild guesses.
Logic dictates no such thing, the barriers up to the very last one had to do with an in-hindsight arrogant belief that the human brain had to be better than any computer at "intelligence" tasks. Obviously that's long gone by now.

The barriers being discussed now have absolutely nothing to do with 'how good can computers become' - we can for the sake of this argument say that given enough effort they could both be oracle-like in accuracy and able to steer for land mines for humans. Given enough technological advancement and effort in engine design something at least close enough to that is surely achievable.

Rather the barrier discussed in this thread is entirely about 'how bad are human players at chess?'. Experience in playing leads a player to come to the conclusion that if they can go ahead a clean piece while still having pawns, it's a 'matter of technique' easy win to finish. It's very difficult to imagine that a top human player could possibly be so bad as not to win a 6 game match at classical time control with an extra knight from the start even against an omnipotent deity able to read their thoughts and steer for where the human will make mistakes. A clean knight advantage is just so durable...
Sorry but the arguments being offered are based on what computers can do TODAY. Which is irrelevant to what computers can do at some distant point in the future where they play perfectly. Who thought 20 years ago that a computer could give a GM a pawn handicap and still win?

Do you REALLY believe that a knight is the asymptote for this? With no evidence other than what is possible today as a basis? I don't know whether a knight is the upper bound or not, and I have said so. I believe anyone that believes that it is is simply making an uninformed/irrational guess based on nothing but history. I don't believe computer chess is at some sort of pinnacle, myself. I have no idea how much further it can go. Only that history suggests than the next breakthrough can and will happen before long.

When you give a knight advantage to a 2000 player, how do you expect him to fare against a 3000 player? Do you believe that knight advantage is "so durable" there? So you give it to a 2800 player, and pit him against a paltry 3800 player. Still "so durable". What about a player that is absolutely perfect? Who knows what that rating will be? Still "so durable"? Yes it is possible, but it is not so probable that one can say "a knight is enough, based on how programs and humans play today, Q.E.D."

That is my point. I'm not taking either size of the knight argument. I am sticking in the middle as "absolutely unknown today."
kbhearn
Posts: 411
Joined: Thu Dec 30, 2010 4:48 am

Re: Something Hikaru Said

Post by kbhearn »

When you give a knight advantage to a 2000 player, how do you expect him to fare against a 3000 player? Do you believe that knight advantage is "so durable" there?
> 75% - I'd happily take an extra knight vs stockfish (which is certainly beyond 3000). a 2000 player reasonably knows to play chess. He should convert a full knight edge regardless of the strength of his opponent most of the time. So yes i believe it's so durable there. I would say it's a feature of the game that it's extraordinarily materialistic and given such a large material edge with no compensation off the start, a reasonable player should expect to win against any opponent.
So you give it to a 2800 player, and pit him against a paltry 3800 player. Still "so durable". What about a player that is absolutely perfect? Who knows what that rating will be? Still "so durable"? Yes it is possible, but it is not so probable that one can say "a knight is enough, based on how programs and humans play today, Q.E.D."
That is my point. I'm not taking either size of the knight argument. I am sticking in the middle as "absolutely unknown today."
I don't know where the asymptote is, that much is true, but i'm pretty sure a knight is beyond it. But then i'm not so sure on for instance double-exchange that by traditional bean counting is beyond a knight. The slow speed at which rooks begin to exert their influence on the game might allow the stronger player to develop sufficient compensation for being two exchanges down to make a game of it. I'd still hope to win it, but it's plausible that it'd make a 'fair' fight against a strong enough opponent.

It is technically true that it's 'unknown' as to how bad humans are at the game but one can still make reasonable guesses about the bounds of what's possible in the confines of the game.
Last edited by kbhearn on Thu Jan 21, 2016 12:05 am, edited 1 time in total.
duncan
Posts: 12038
Joined: Mon Jul 07, 2008 10:50 pm

Re: Something Hikaru Said

Post by duncan »

bob wrote: That is my point. I'm not taking either size of the knight argument. I am sticking in the middle as "absolutely unknown today."
I value what you have to say and you have said the following. so I assume in the above you are speaking scientifically.

speaking personally about your feeling about the matter though, you have your suspicions/opinions.



I suspect it might be beyond a knight, myself, knowing the human propensity for making inexact moves


What is the max material needed as an advantage for a 2800 player to beat a 60,000 player? Absolutely no idea for me. But I suspect a knight won't even be close.


And for me, 48,000 would be an acceptable estimate, because it is only 45,000 Elo above the best humans. Would that be enough to win a knight odds game from the weaker side? Every time? I'd suspect so, not that we will ever see that sort of perfect play from a computer.

I suspect that a knight will not be enough in another 50 years. But that is an opinion only.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Something Hikaru Said

Post by bob »

Laskos wrote:
bob wrote:
700-1500 Elo is certainly meaningless. Where does that come from? 20 years ago the assumption was that 2800 was the upper bound on Elo. That seems to have bitten the big banana. The only thing that bounds Elo is that the best player will be hard-pressed to get more than 800 above the second-best player. But then the second best can get to 800 below the 3rd.
Well, I will show how that is "meaningless". Some 2-3 years ago I computed the estimate of the rating of the perfect engine as limiting ELO of Houdini 3 to infinite number of nodes (time control). Duncan refers to that post of mine. Today I repeated with thousands of games this estimate with Komodo 9.3 to infinite time control (doubling in time). I guess you consider ELO gain per doubling as "close to 70 ELO points", which is totally misleading, and the diminishing returns are thought by you as "hard to measure".

My today's tests:
First doubling: 10s+0.1s vs 5s+0.05s -- 148 ELO points
Second doubling: 20s + 0.2s vs 10s+0.1s -- 128 ELO points
Third doubling: 40s+0.4s vs 20s+0.2s -- 110 ELO points

Other data:
Close to eighth doubling -- CEGT 40/20' -- 65 ELO points
Close to ninth doubling -- CCRL 40/40' -- 55 ELO points

Diminishing returns are very visible here. I fitted these results with the relevant curve. Here is the plot of ELO gain per doubling in time:
Image

The red dots are not data, they are predictions, first for Larry's 45'+15'' on 24 cores level against humans in odds matches, for which Larry got as Komodo performing at 3250 FIDE ELO level, second red dot shows TCEC level of 150'+30'' on 24 faster cores. One sees that at those high levels the gain per doubling is below 40 ELO points.

The fit: the fitting curve chosen is relevant as I show. It is a/(1+b*(number of doublings)^c). {a,b,c} are parameters to fit, and {c} is the relevant exponent here. If {c} is between 0 and 1, then we do have diminishing returns, but the total ELO is unbounded for higher and higher number of doublings, therefore the rating of perfect engine is undetermined and high. If c>1, then the perfect engine has a definite limited rating which can be derived.

It turns out that the best fit is c=1.56, which is significantly larger than 1, and the ELO of the perfect engine can be computed and is not very high. I actually derive this, it is not assumed.

Do you have a more relevant fitting expression? I assume only that diminishing gains are going to 0 gain per doubling to infinite number of doublings, and this is related to the draw ratio going to 100% to infinite time control (doublings). Do you have a more plausible model which shows that the draw ratio has in fact lower limit than 100%?

Having the fit, I can compute the ELO of the perfect engine by summing up all gains from doublings starting from established by Larry FIDE ELO 3250 (close to 13th doubling in time). It is ~1300 ELO points above Larry's 24 core Komodo. Therefore, Komodo 9.3 shows a FIDE ELO of the perfect engines at about 3250+1300 = 4550 ELO points. CCRL rating would be 100-200 ELO points higher, as their rating is computer rating. And very close to 4800 CCRL I got 2-3 years ago with Houdini 3.

As you see, 700-1500 ELO points estimate for improvement over Komodo I gave previously is not meaningless at all, my model here is simple, robust, and consistent with earlier results (including one or two by Don Dailey). If you want to dismiss it, it's surely not by "we don't know" mantra, because it seems it is mostly you who "doesn't know".
Again, your "diminishing returns" applies to programs of today and earlier. Forget the "doubling" stuff, we are not talking about doubling. We are talking about an infinite improvement, all the way to perfect play. If you take programs from 1970, you would get a completely different perspective vis a vis "doubling". But we are not in 1970. And 100 years in the future, we won't be stuck with 2016 era programs.

I don't know how to predict what will happen as we approach perfection. I only know that if you wait long enough chess will eventually get there, assuming the sun continues to burn at the present rate, etc... I don't see any way to project to perfect play since we have no idea of what perfect play looks like..
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Something Hikaru Said

Post by bob »

Jesse Gersenson wrote:
bob wrote: I did a quick online search and did not find any results produced after the game solution was announced...
bob wrote: not AFTER they finished the final endgame tables that let them play perfectly. He himself had branded it "unbeatable".
My mistake, I'd overlooked the stipulation of 'after game solution was announced'.

Checkers was 'solved' in 2007(?). Tinsley was already 65 when he beat Chinook in their first match (1992) and was 68 when he played it again, drawing 6 games before retreating to die.
I knew him well. The world checker hall of fame is in Petal Mississippi, about 8 miles from my old office at the University of Southern Mississippi. Marion and Charles Walker (owned the hall of fame) were good friends and they used to come out to my office whenever Marion was in town, always wanting to play skittles games with Cray Blitz. Tinsley was a pretty fair chess player, although not up to playing CB in 5-10 minute game speeds.

Used to enjoy sitting around a table and talking with him, and with a couple of the checker championship challengers he and Charles would drag out to my office to play chess on off-days...

He declared the final Chinook as absolutely unbeatable, which it was thanks to the endgame tables. He also didn't think he could play flawless checkers game in and game out, and would ultimately lose to Chinook by making a mistake every now and then even though he was incredibly accurate.