Something Hikaru Said

[syzygy]

Without handicap ? 1 on 5000 maybe.

Yes, without handicap.

So 1 in 5000 could be an estimate for the number of Komodo half-point errors.

Now consider a knight odds game where the human makes a half-point error but still beats Komodo. That means Komodo made a half-point error as well. So that should happen only 1 in 5000 games where the human made an error. That means improving Komodo further should only make a difference in those 1 in 5000 games (of the already few games where the human made an error). In other words, there is hardly anything to gain at knight odds by further improving the engine.

This is of course not a mathematical proof, but it is an argument that would seem to have some force.

Don't make the mistake of assuming that Komodo of today is playing near-perfect chess. Also don't assume that komodo 100 or 1000 years in the future will only gain 200 Elo or so...

I have extensively discussed why further improvements will hardly gain anything at knight odds.

ALL based on current Komodo vs GM results....

Exactly! On the basis of solid numbers I have given arguments why further improvements will hardly gain anything at knight odds. Even if further improvements gain 1000s of Elo in normal chess, at knight odds engines will hardly profit from it.

[bob]

Without handicap ? 1 on 5000 maybe.

Yes, without handicap.

So 1 in 5000 could be an estimate for the number of Komodo half-point errors.

Now consider a knight odds game where the human makes a half-point error but still beats Komodo. That means Komodo made a half-point error as well. So that should happen only 1 in 5000 games where the human made an error. That means improving Komodo further should only make a difference in those 1 in 5000 games (of the already few games where the human made an error). In other words, there is hardly anything to gain at knight odds by further improving the engine.

This is of course not a mathematical proof, but it is an argument that would seem to have some force.

Don't make the mistake of assuming that Komodo of today is playing near-perfect chess. Also don't assume that komodo 100 or 1000 years in the future will only gain 200 Elo or so...

I have extensively discussed why further improvements will hardly gain anything at knight odds.

ALL based on current Komodo vs GM results....

Exactly! On the basis of solid numbers I have given arguments why further improvements will hardly gain anything at knight odds. Even if further improvements gain 1000s of Elo in normal chess, at knight odds engines will hardly profit from it.

As I said, with NO data about the issue of 32 piece endgame type skill. What Komodo does today is irrelevant. What is important is what will happen at some point in the distant future? And the sample is based upon GM/program play today. GMs are not going to improve tactically.

The only extrapolated data I saw was 800 real elo translates to about 1/4 of that with knight odds. What happens with 10,000 Elo?

[Laskos]

Without handicap ? 1 on 5000 maybe.

Yes, without handicap.

So 1 in 5000 could be an estimate for the number of Komodo half-point errors.

Not necessarily. Maybe Komodo is making 10 half- or full- point errors every 60 moves, so almost every game it's making such errors. Only that human is making 20 such errors every 60 moves. Almost every game too.

Well, you're right that it is not correct to equate the Komodo half-point error rate to the win rate of a human GM against Komodo without handicap. Clearly the human will often not be able to convert such an error into a win.

Still the reasoning seems to be valid.

Instead of referring to 1 in 5000 as the Komodo half-point error rate, I should have called it the rate at which Komodo loses theoretically drawn positions against a 2500 Elo GM. (This is still a tricky generalisation, of course. There are huge differences between "theoretically drawn positions" once imperfect play of engines and GMs is involved.)

Let's take X for the fraction of knight odds games that an 2500 Elo GM plays flawlessly (i.e. keeping a winning position from start to end).

The fraction of knight odds games where Komodo reaches a theoretically drawn (or winning) position is then 1-X. Of those games, the human might still win 1 in 5000.

So the fraction of games won by the human is X + (1-X) * (1/5000).

Taking Larry's numbers, we get X + (1-X) * (1/5000) = 0.90. (The games that are not won should be mostly draws, so the score is indeed 95%.)

This gives X = 0.89998.

Now replace Komodo by an engine that plays perfect chess. The fraction of games won by the 2500 Elo GM is now "only" X. So the 2500 Elo GM will "only" score 94.998% or so.

This does not take into account the possibility that a stronger engine might make it harder for the human to play a perfect knight odds game. But I fail to see how e.g. speeding up Komodo by a factor of 1 mlllion would make it more difficult. Special anti-human tricks on the other hand should be able to make a (limited) difference.

Yes, that holds, to shake it I have to unnaturally complicate the things with "probability that Komodo punishes the error" and such. But I would give a a model which I consider more plausible on empirical grounds:

In Knight odds chess the human GM game-changing error per move is much lower than in normal chess, I will take it conveniently at 0.4%. Komodo, starting with a lost position, can err only by not punishing the error of the human. Say Komodo punishes 50% of human game-changing errors. The total rate of Komodo game-changing errors is 0.4%*0.5=0.2% per move. The average game length against Komodo is 60 moves.

Now take the perfect engine instead of Komodo. First, it makes a human blunder 0.6% of the moves (some increase in blunder rate because of trickier positions), punishes all blunders and lengthens the game to 90 moves (some increase in game length). These small incremental factors add up to make a large difference:

Against Komodo human wins (1-0.004*0.5)^60 ~ 90% of the games, the rest being mostly draws, as before per Larry's 95%.
Against perfect engine human wins (1-0.006*1)^90 ~ 58% of the games, with the score something like 75% for human.

There is a factor of 4 now between non-won by human games in the 2 cases, a large difference. The difference of 500 ELO points against Komodo becomes 200 ELO points against perfect engine.

Check for consistency: consider an ELO 2000 player. He has 50% score against Komodo at Knight odds. I will try to describe this outcome simply by his game-changing blunder rate of 4% per move in Knight odds chess.

He has (1-0.04*0.5)^60 ~ 30% win rate against Komodo at Knight odds for a score of 50%.
Against the perfect engine he has (1-0.04*1)^90 ~ 2.5% win rate for a score of 15%, so 300 ELO points behind the perfect engine at Knight odds.

The result is consistent with assigned 2000 rating if we assign 2500 to the GM and derive 2300 for perfect engine at Knight odds, which improved at Knight odds compared to Komodo by 300 ELO points in both cases.

This model assumes some incremental, cumulative small differences between perfect engine and Komodo from the point of view of a human player. On grounds of evidence and plausibility, I would stand more by it, but my model sure is ugly.

[duncan]

I don't know that the machine will be able to win, but there is absolutely ZERO evidence to support that it will always lose. Absolutely ZERO. To make such a claim, there must be something to support it.

what is the reason that this statement of larry would not be evidence that the machine will lose to knight handicap ?

is it because he has not proven that K/C ratio cannot be 64%, or something else. ?

http://www.talkchess.com/forum/viewtopi ... 49&t=58846

Let's say that there is some handicap that would produce an even score in a serious match between Komodo and Magnus Carlsen. I would estimate that this handicap would be in the 1 to 1.5 pawn range based on Komodo's eval after a long think. Let's say 1.25. Now both Komodo and Carlsen make errors of some average magnitude. We'll call Carlsen's error rate C, and Komodo's K. I think it's pretty obvious that K is much less than C, let's say K = .4xC. If a future engine drops the error rate to zero, then C - K increaases to 5/3 of it's former value, so the proper handicap should also increase in that ratio. That would put it at 208, a bit over two pawns but way below the roughly 3.5 value of knight odds. Of course there is a lot of uncertainty in the above, but I don't think the estimate would be way off. In order for the estimate to reach knight odds, the K/C ratio would have to be about 64%, which does not seem plausible to me

How can you prove anything with no way to verify? He said something like the thought Komodo was within 1 pawn of playing perfectly. Not from the games I have seen. It is very strong, but it is FAR short of playing perfectly, based on analysis of output I have seen... And that is just today, of course. What happens in the future, or even beyond the future with the hypothetical access to 31 piece endgame tables?

If you backed up in time, say 20 years, would anyone be claiming that a computer could give a GM anything and still win? Not in any discussion I ever took part in, even at ACM/ICGA/ICCA chess events. So the bar has moved higher in 20 years. Is there empirical evidence that suggests that we are "at the max point of computer chess strength" already?

10-20 years ago we had lots of claims that computers were GMs when they were not. Now we seem to be at the point where some think the computer plays almost perfect chess, which it can't. And then some also assume that a human can play a knight odds game perfectly. He can't play a non-odds game perfectly so what suggests that taking away one of the opponent's pieces suddenly increases his skill level so dramatically?

I don't know what the max odds will be, and don't really care either. But to claim there is some asymptote that can't be crossed seems just a tae premature when there is no evidence to support this.

(1)In my rather limited understanding of all this I believe (rather than arguing Komodo was within 1 pawn of playing perfectly.) that larry was arguing from a current komodo 1.25 advantage over super gm to a future 32 piece perfect engine, the advantage would only increase to about 2.2 advantage.

his argument was based that in a perfect engine C - K increases only to 5/3 of it's former value.


what was the fallacy in his argument.?

(2)do you know how about many moves it takes a gm to convert a knight handicap from 3 to 4 advantage?

(3)let's say a gm can convert a knight handicap from 3 to 4 in 9 moves. would a future 60,000 elo computer have to be able to get the score from 3 to 2 in less than 9 moves to win.?

[Jesse Gersenson]

ALL based on current Komodo vs GM results....

No, it's all based on results against players rated 2080-2200. Komodo hasn't played GM's at knight odds with a 45m+15s time control.

Larry took Komodo's performance rating of 'slightly less than 2000', giving knight odds against players rated 2080-2200, and used the ELO scale to deduce the performance against a player rated 2500.

This is flawed if we agree the knight handicap will be more advantageous for the 2500 than it was for the 2150, i.e. Komodo's performance rating will be less than 'slightly less than 2000' when playing a 2500 GM. (There's no evidence showing it's performance rating wouldn't be 1300?!)

[bob]

Without handicap ? 1 on 5000 maybe.

Yes, without handicap.

So 1 in 5000 could be an estimate for the number of Komodo half-point errors.

Not necessarily. Maybe Komodo is making 10 half- or full- point errors every 60 moves, so almost every game it's making such errors. Only that human is making 20 such errors every 60 moves. Almost every game too.

Well, you're right that it is not correct to equate the Komodo half-point error rate to the win rate of a human GM against Komodo without handicap. Clearly the human will often not be able to convert such an error into a win.

Still the reasoning seems to be valid.

Instead of referring to 1 in 5000 as the Komodo half-point error rate, I should have called it the rate at which Komodo loses theoretically drawn positions against a 2500 Elo GM. (This is still a tricky generalisation, of course. There are huge differences between "theoretically drawn positions" once imperfect play of engines and GMs is involved.)

Let's take X for the fraction of knight odds games that an 2500 Elo GM plays flawlessly (i.e. keeping a winning position from start to end).

The fraction of knight odds games where Komodo reaches a theoretically drawn (or winning) position is then 1-X. Of those games, the human might still win 1 in 5000.

So the fraction of games won by the human is X + (1-X) * (1/5000).

Taking Larry's numbers, we get X + (1-X) * (1/5000) = 0.90. (The games that are not won should be mostly draws, so the score is indeed 95%.)

This gives X = 0.89998.

Now replace Komodo by an engine that plays perfect chess. The fraction of games won by the 2500 Elo GM is now "only" X. So the 2500 Elo GM will "only" score 94.998% or so.

This does not take into account the possibility that a stronger engine might make it harder for the human to play a perfect knight odds game. But I fail to see how e.g. speeding up Komodo by a factor of 1 mlllion would make it more difficult. Special anti-human tricks on the other hand should be able to make a (limited) difference.

None of this seems reasonable to me. The percentage of GM errors, for example. A GM makes far FEWER errors against a weak program than against a strong program, but not because he actually makes fewer errors, but because the opponent doesn't notice them and doesn't punish them.

I don't see any way to extrapolate an error rate against "god" (31 piece EGTBs) until we actually get there... It might turn out that EVERY move he makes is an error when the opponent makes absolutely no mistakes in return and misses no mistake the GM makes when he does make one.

[bob]

ALL based on current Komodo vs GM results....

No, it's all based on results against players rated 2080-2200. Komodo hasn't played GM's at knight odds with a 45m+15s time control.

Larry took Komodo's performance rating of 'slightly less than 2000', giving knight odds against players rated 2080-2200, and used the ELO scale to deduce the performance against a player rated 2500.

This is flawed if we agree the knight handicap will be more advantageous for the 2500 than it was for the 2150, i.e. Komodo's performance rating will be less than 'slightly less than 2000' when playing a 2500 GM. (There's no evidence showing it's performance rating wouldn't be 1300?!)

There's no real evidence showing it would be better or worse, when it plays perfectly. Hence my concern that these extrapolations are not even close to "SWAGs"...

I would bet the human GM errs FAR more than the numbers being quoted, it is just that today's programs don't spot or punish most of the errors. As a simple example, look at all the computer cooks to simple chess endgame studies in books such as Fine's "Basic Chess Endings". A book that was taken as gospel in the 40's and is now known to be riddled with errors nobody knew about until computers with EGTBs came along.

The only errors WE see are the errors we recognize. That doesn't mean that those are ALL the errors made by a human GM in the game. Can't count what you can't recognize. But a 32 piece EGTB program would recognize EVERY single error.. How many that is is idle speculation, however. I simply don't believe it is possible to conclude that a knight handicap is too much for such a program to win. It might be, but there is no evidence as of yet to suggest that is an accurate upper bound. One must exist, but where it is is pure conjecture at the moment. I suspect it might be beyond a knight, myself, knowing the human propensity for making inexact moves.

[Uri Blass]

None of this seems reasonable to me. The percentage of GM errors, for example. A GM makes far FEWER errors against a weak program than against a strong program, but not because he actually makes fewer errors, but because the opponent doesn't notice them and doesn't punish them.

I disagree here.
If the opponent does not play well it is easier not to make mistakes.

I clearly have games against humans when I did no significant mistakes based on computer analysis(no move reduce the evaluation by more than 0.2 pawn).

It is not because I am so strong but because it is easier not to make mistakes when the opponent does not play well.
If the opponent play well I expect myself to do more mistakes.

Uri

[bob]

None of this seems reasonable to me. The percentage of GM errors, for example. A GM makes far FEWER errors against a weak program than against a strong program, but not because he actually makes fewer errors, but because the opponent doesn't notice them and doesn't punish them.

I disagree here.
If the opponent does not play well it is easier not to make mistakes.

I clearly have games against humans when I did no significant mistakes based on computer analysis(no move reduce the evaluation by more than 0.2 pawn).

It is not because I am so strong but because it is easier not to make mistakes when the opponent does not play well.
If the opponent play well I expect myself to do more mistakes.

Uri

"no significant mistakes based on computer analysis" is meaningless when we are talking about PERFECT computer play. ANY mistake will be significant there. This extrapolation about what happens today is meaningless when we talk about a perfect chess opponent.

[Laskos]

None of this seems reasonable to me. The percentage of GM errors, for example. A GM makes far FEWER errors against a weak program than against a strong program, but not because he actually makes fewer errors, but because the opponent doesn't notice them and doesn't punish them.

I disagree here.
If the opponent does not play well it is easier not to make mistakes.

I clearly have games against humans when I did no significant mistakes based on computer analysis(no move reduce the evaluation by more than 0.2 pawn).

It is not because I am so strong but because it is easier not to make mistakes when the opponent does not play well.
If the opponent play well I expect myself to do more mistakes.

Uri

"no significant mistakes based on computer analysis" is meaningless when we are talking about PERFECT computer play. ANY mistake will be significant there. This extrapolation about what happens today is meaningless when we talk about a perfect chess opponent.

Some extrapolations are meaningless some are not. That perfect engine is in the range 700-1500 ELO points stronger than Komodo is not meaningless. That only of fraction of that will go to improvement of Knight-odds chess is not meaningless. That this fraction is probably less than 50% is not meaningless.

Some other arguments are not meaningless. That Nakamura said what he said is not meaningless. It seems you promote complete ignorance on a subject where we have lots of empirical data and commons sense.