Hikaru vs. bots

[Uri Blass]

My fide rating today is 1963
My chess.com rating in blitz is 1954
My maximal fide rating was 2051

[Madeleine Birchfield]

Today chess.com had an event in which participants played against many of their chess bots, which are basically various Komodo (version 13.3 I think) Skill levels modified in various ways. Levels 19 and below use evals from NNs that are designed to make them more "human", but the bots rated over 2400 use the Komode evals with Skill levels 20 thru 22 and a few parameter mods. I watched most of the games by Hikaru Nakamura against the bots. There was no time limit per game, but as prizes were based on how many bots were beaten, there was an incentive to play quickly and to resign bad or drawn positions; I would estimate the rate of play was similar to 3' + 2" games, with the engine playing at hyper-bullet speed. As you would expect, Hikaru beat all the bots below level 20, though in a couple cases he didn't win on the first try, at least at level 19. But he had a tough time with the Skill 20 bots; I think he eventually beat them all, but probably lost more than he won, even though they were only doing seven ply searches and were slightly weakened by other parameter changes. Against the level 21 bots, he could not score against one of them, and didn't win right away against others. Against level 22 bots (9 ply), he beat the "Morphy" bot, but lost all games to his own Hikaru bot! Overall the bots gave him a tougher time than I would have predicted. It seems that his closest match was level 20.

Engines with handcrafted evaluations do not play in the same way as engines with neural networks do, so I am not surprised that Hikaru Nakamura had a hard time with the bots based upon Komodo 13.3.

[lkaufman]

Today chess.com had an event in which participants played against many of their chess bots, which are basically various Komodo (version 13.3 I think) Skill levels modified in various ways. Levels 19 and below use evals from NNs that are designed to make them more "human", but the bots rated over 2400 use the Komode evals with Skill levels 20 thru 22 and a few parameter mods. I watched most of the games by Hikaru Nakamura against the bots. There was no time limit per game, but as prizes were based on how many bots were beaten, there was an incentive to play quickly and to resign bad or drawn positions; I would estimate the rate of play was similar to 3' + 2" games, with the engine playing at hyper-bullet speed. As you would expect, Hikaru beat all the bots below level 20, though in a couple cases he didn't win on the first try, at least at level 19. But he had a tough time with the Skill 20 bots; I think he eventually beat them all, but probably lost more than he won, even though they were only doing seven ply searches and were slightly weakened by other parameter changes. Against the level 21 bots, he could not score against one of them, and didn't win right away against others. Against level 22 bots (9 ply), he beat the "Morphy" bot, but lost all games to his own Hikaru bot! Overall the bots gave him a tougher time than I would have predicted. It seems that his closest match was level 20.

Engines with handcrafted evaluations do not play in the same way as engines with neural networks do, so I am not surprised that Hikaru Nakamura had a hard time with the bots based upon Komodo 13.3.

Are you saying that he would do better against similarly rated bots that used NNs? Actually the weaker bots (those with est. ratings below 2400) did use tiny NNs. He did well against them, but they were weak intentionally. The skill levels on Dragon, using NNUE, are about equal to those on Komodo in direct play, so you would predict that Hikaru would score better against the same skill levels on Dragon? I wouldn't predict that.

[Milos]

I understand that draw or loss was the same for hikaru.

It is not the case in case hikaru had to play for a CCRL rating list.

Another advantage hikaru had for the ccrl rating list is that he could think on the opponent time when he cannot do it when the opponents play at hyper bullet speed(I know CCRL use no pondering but still you cannot force the opponent not to ponder in case the opponent is not an engine).

For the time control I understand that the time control in CCRL blitz is
equivalent to 2'+1" on an Intel i7-4770K

I wonder if you use the same hardware or maybe faster hardware.

Hardware for this was calibrated based on an old 16 core Xeon, presumably somewhat slower than the CCRL reference machine. So really I overrated the skill levels slightly. If Hikaru (or any human) did play games under CCRL conditions, presumably Ponder would be turned on for the machine to be fair, since it can't be turned off for the human. It's true that he got nothing for draws, so perhaps if we allow for this his fair opponent was between level 20 and 21, still only something like 2050 CCRL blitz.

I tried to play against Nero6.1 that has ccrl rating of 1447 at 2+1 and won with white and I believe that even at this time control I am not weaker than Nero and probably better than it.
I am sure nakamura is more than 600 elo better than me at 2+1 so he has to get more than 2050 CCRL rating because I guess I can get more than 1450 in ccrl 2+1 inspite of not being a strong blitz chess player.

Human Elo scaling is not the same as engine Elo scaling (not just offset but actual value of Elo point is different). So the fact that Naka is 600Elo stronger than you in "human" Elo, might mean only 300 "engine" Elo.
At 2+1 Naka is probably stronger than 2050 CCRL but not by much, maybe around 2150-2200 Elo. Chess.com ratings in blitz are certainly overinflated. Probably difference of 1000Elo on Chess.com is less than 500Elo on CCRL.

[lkaufman]

I understand that draw or loss was the same for hikaru.

It is not the case in case hikaru had to play for a CCRL rating list.

Another advantage hikaru had for the ccrl rating list is that he could think on the opponent time when he cannot do it when the opponents play at hyper bullet speed(I know CCRL use no pondering but still you cannot force the opponent not to ponder in case the opponent is not an engine).

For the time control I understand that the time control in CCRL blitz is
equivalent to 2'+1" on an Intel i7-4770K

I wonder if you use the same hardware or maybe faster hardware.

Hardware for this was calibrated based on an old 16 core Xeon, presumably somewhat slower than the CCRL reference machine. So really I overrated the skill levels slightly. If Hikaru (or any human) did play games under CCRL conditions, presumably Ponder would be turned on for the machine to be fair, since it can't be turned off for the human. It's true that he got nothing for draws, so perhaps if we allow for this his fair opponent was between level 20 and 21, still only something like 2050 CCRL blitz.

I tried to play against Nero6.1 that has ccrl rating of 1447 at 2+1 and won with white and I believe that even at this time control I am not weaker than Nero and probably better than it.
I am sure nakamura is more than 600 elo better than me at 2+1 so he has to get more than 2050 CCRL rating because I guess I can get more than 1450 in ccrl 2+1 inspite of not being a strong blitz chess player.

Human Elo scaling is not the same as engine Elo scaling (not just offset but actual value of Elo point is different). So the fact that Naka is 600Elo stronger than you in "human" Elo, might mean only 300 "engine" Elo.
At 2+1 Naka is probably stronger than 2050 CCRL but not by much, maybe around 2150-2200 Elo. Chess.com ratings in blitz are certainly overinflated. Probably difference of 1000Elo on Chess.com is less than 500Elo on CCRL.

Certainly Human elo differences may differ from engine Elo differences, as you say. However, the accepted opinion, especially based on the studies of Kai Laskos, is that human elo differences are LESS than engine elo differences. To be fair, he and others were talking about FIDE ratings, and chess.com ratings are more spread out than FIDE ratings, maybe by 5/4 or so. But Kai estimated that FIDE rating differences were 2/3 (or even a bit less, I forget the exact figure) of engine rating differences, so even if you multiply 2/3 by 5/4 you get 5/6, well below 1. So unless his (and other) studies on this issue were way off the mark, chess.com rating differences should still be slightly less than CCRL differences, surely not significantly more. I'm suspecting the problem has more to do with specific engines, for example I keep losing at 2' + 1" to "Irina 0.15", which has a similar rating to the Nero engine mentioned above. OK, I'm 73 and move too slowly for 2' +1" chess, but I would be very surprised if Uri could beat "Irina" at that time control. You are probably right that Naka would be at least 2150 playing CCRL engines at 2' + 1", maybe crippled Komodos are relatively harder for humans than weak normal engines of the same "rating".

[Milos]

I understand that draw or loss was the same for hikaru.

It is not the case in case hikaru had to play for a CCRL rating list.

Another advantage hikaru had for the ccrl rating list is that he could think on the opponent time when he cannot do it when the opponents play at hyper bullet speed(I know CCRL use no pondering but still you cannot force the opponent not to ponder in case the opponent is not an engine).

For the time control I understand that the time control in CCRL blitz is
equivalent to 2'+1" on an Intel i7-4770K

I wonder if you use the same hardware or maybe faster hardware.

Hardware for this was calibrated based on an old 16 core Xeon, presumably somewhat slower than the CCRL reference machine. So really I overrated the skill levels slightly. If Hikaru (or any human) did play games under CCRL conditions, presumably Ponder would be turned on for the machine to be fair, since it can't be turned off for the human. It's true that he got nothing for draws, so perhaps if we allow for this his fair opponent was between level 20 and 21, still only something like 2050 CCRL blitz.

I tried to play against Nero6.1 that has ccrl rating of 1447 at 2+1 and won with white and I believe that even at this time control I am not weaker than Nero and probably better than it.
I am sure nakamura is more than 600 elo better than me at 2+1 so he has to get more than 2050 CCRL rating because I guess I can get more than 1450 in ccrl 2+1 inspite of not being a strong blitz chess player.

Human Elo scaling is not the same as engine Elo scaling (not just offset but actual value of Elo point is different). So the fact that Naka is 600Elo stronger than you in "human" Elo, might mean only 300 "engine" Elo.
At 2+1 Naka is probably stronger than 2050 CCRL but not by much, maybe around 2150-2200 Elo. Chess.com ratings in blitz are certainly overinflated. Probably difference of 1000Elo on Chess.com is less than 500Elo on CCRL.

Certainly Human elo differences may differ from engine Elo differences, as you say. However, the accepted opinion, especially based on the studies of Kai Laskos, is that human elo differences are LESS than engine elo differences. To be fair, he and others were talking about FIDE ratings, and chess.com ratings are more spread out than FIDE ratings, maybe by 5/4 or so. But Kai estimated that FIDE rating differences were 2/3 (or even a bit less, I forget the exact figure) of engine rating differences, so even if you multiply 2/3 by 5/4 you get 5/6, well below 1. So unless his (and other) studies on this issue were way off the mark, chess.com rating differences should still be slightly less than CCRL differences, surely not significantly more. I'm suspecting the problem has more to do with specific engines, for example I keep losing at 2' + 1" to "Irina 0.15", which has a similar rating to the Nero engine mentioned above. OK, I'm 73 and move too slowly for 2' +1" chess, but I would be very surprised if Uri could beat "Irina" at that time control. You are probably right that Naka would be at least 2150 playing CCRL engines at 2' + 1", maybe crippled Komodos are relatively harder for humans than weak normal engines of the same "rating".

I don't think that FIDE Elo scale is compressed compared to CCRL Elo scale. Actually there are a lot of indications for the opposite. Don't really remember Kai's study, but somehow I think it was about something different. Like that to humans very strong engines are all look alike, i.e. for humans there is very little difference playing engine of 2900Elo CCRL and 3400Elo CCRL. However, the opposite is also true, for strong engines there is very little difference playing human super GMs and 2200Elo FIDE rated players. But this is all due to the breaking of Elo model once difference between opponents becomes larger than 500Elo.

[Vinvin]

For the record, around year 2000, Crafty was already a bit stronger than strong GMs in blitz, so, 20 years later, one can expect strong engines (on current hardware) to be 1000 to 1500 Elo stronger than current top GMs.

"Crafty on ICC vs GMs" from the archives :
https://www.stmintz.com/ccc/index.php?id=174616
https://www.stmintz.com/ccc/index.php?id=154243
https://www.stmintz.com/ccc/index.php?id=254133
https://www.stmintz.com/ccc/index.php?id=355397

[Vinvin]

more ...
In 1997, Rebel beat Yusupov 6-1 in blitz (5+0) : http://www.rebel.nl/june2.htm
In 1998, Rebel beat Anand 4.5-1.5 in long blitz (5+5 and 15+0) : http://www.rebel.nl/july21.htm

[mwyoung]

Lets go back before the year 1997. And look at Fritz 2. Computer were beating the best players for many years. Stupid, small, and simple engines on slow single core CPU's

Code: Select all

Intel® Pentium® Processor	200-75 MHz	Mar-94	0.6 and 0.35 micron	3.3 million	8 kB L1 Cache	4 GB	66 MHz
60 MHz
50	Desktops

World Chess Express Challenge 1994. Quotes-

‘It won’t win, but it won’t
come last,’ promises Fred Friedel, a founder of the company which is developing
the program, called Fritz3.

Freidel hopes that the program will be able to hold its own against
humans, although he is uncertain whether a computer will ever become the
best player in the world. ‘If one did, you would just exclude computers
from competing in tournaments,’ he says. ‘The humans play on.’

‘In fast chess, you get run over by (computer) tactics, because it’s
so easy to overlook something,’ says Friedel. ‘But in classic chess, with
more time, every grandmaster seems to have some built-in ability to check
beyond the horizon for tactical traps. If they can see as far as the computer,
it just becomes plan against plan. And there humans are superior.’

The name of the game, at least at this stage, is processing speed. ‘Every
time I get a fivefold increase in computing power, the program can look
one move deeper in the same time,’ says Friedel. ‘So if you give me something
20 times more powerful, it can look two moves deeper.’ The Fritz programs
spend perhaps half the time searching all available moves up to a depth
of about seven moves, and may then take the most promising variations 20
or 30 moves deep, depending on the situation. But even this is not always
far enough to compete with the grandmasters.

According to the official Elo chess rating, Fritz2 is already ranked
higher than more than 99 per cent of the players in the world.

But could a computer ever beat a human world champion? Probably not,
thinks Friedel. ‘This is the fascination for me,’ he says. ‘Although you
get so close and can beat 99.999 per cent of the world, the remaining handful
of top players remains unassailable. The work has just begun.’

Code: Select all

Chess: Kasparov is the only man able to beat Fritz
WILLIAM HARTSTON
Tuesday 24 May 1994 00:02 
TWENTY YEARS ago, computer programmers were saying that the world chess champion would lose to a machine within 10 years. Grandmasters were saying 10 years ago that no computer would ever play chess at their level. Both were wrong.

High-level human thought in chess is based on strategic concepts derived from experience. When putting black pawns on c5, d6 and e5, for example, we learn to judge whether the hole at d5 is a serious liability. We do not need to calculate 10 or 20 moves ahead until an enemy piece lands there. Which is fortunate, because our minds could never scale such mountains of calculation.

Computers, while unable to simulate our grasp of strategic ideas, are very good at doing the sums. But even thinking at a rate of a million moves a second, machines are still making errors that are elementary by human standards.

When Intel sponsored the World Chess Express Challenge in Munich last Friday, they could never have hoped for such a good advertisement for their high-speed Pentium processor. It turned a good computer - Fritz 2 - into a world beater.

In a day of five-minute games, it beat Kasparov, Short, Anand, Kramnik and Gelfand. Finally, Fritz 2 and Kasparov shared first place. Kasparov won the play-off by 4-1, but it is only a matter of time before the machine overtakes even him.

Tomorrow we shall look at Fritz's win against Nigel Short. Meanwhile, here's a sobering thought: Fritz 2 can examine more positions during a single game than a human being can in an entire lifetime.

[lkaufman]

Lets go back before the year 1997. And look at Fritz 2. Computer were beating the best players for many years. Stupid, small, and simple engines on slow single core CPU's

Code: Select all

Intel® Pentium® Processor	200-75 MHz	Mar-94	0.6 and 0.35 micron	3.3 million	8 kB L1 Cache	4 GB	66 MHz
60 MHz
50	Desktops

World Chess Express Challenge 1994. Quotes-

‘It won’t win, but it won’t
come last,’ promises Fred Friedel, a founder of the company which is developing
the program, called Fritz3.

Freidel hopes that the program will be able to hold its own against
humans, although he is uncertain whether a computer will ever become the
best player in the world. ‘If one did, you would just exclude computers
from competing in tournaments,’ he says. ‘The humans play on.’

‘In fast chess, you get run over by (computer) tactics, because it’s
so easy to overlook something,’ says Friedel. ‘But in classic chess, with
more time, every grandmaster seems to have some built-in ability to check
beyond the horizon for tactical traps. If they can see as far as the computer,
it just becomes plan against plan. And there humans are superior.’

The name of the game, at least at this stage, is processing speed. ‘Every
time I get a fivefold increase in computing power, the program can look
one move deeper in the same time,’ says Friedel. ‘So if you give me something
20 times more powerful, it can look two moves deeper.’ The Fritz programs
spend perhaps half the time searching all available moves up to a depth
of about seven moves, and may then take the most promising variations 20
or 30 moves deep, depending on the situation. But even this is not always
far enough to compete with the grandmasters.

According to the official Elo chess rating, Fritz2 is already ranked
higher than more than 99 per cent of the players in the world.

But could a computer ever beat a human world champion? Probably not,
thinks Friedel. ‘This is the fascination for me,’ he says. ‘Although you
get so close and can beat 99.999 per cent of the world, the remaining handful
of top players remains unassailable. The work has just begun.’

Code: Select all

Chess: Kasparov is the only man able to beat Fritz
WILLIAM HARTSTON
Tuesday 24 May 1994 00:02 
TWENTY YEARS ago, computer programmers were saying that the world chess champion would lose to a machine within 10 years. Grandmasters were saying 10 years ago that no computer would ever play chess at their level. Both were wrong.

High-level human thought in chess is based on strategic concepts derived from experience. When putting black pawns on c5, d6 and e5, for example, we learn to judge whether the hole at d5 is a serious liability. We do not need to calculate 10 or 20 moves ahead until an enemy piece lands there. Which is fortunate, because our minds could never scale such mountains of calculation.

Computers, while unable to simulate our grasp of strategic ideas, are very good at doing the sums. But even thinking at a rate of a million moves a second, machines are still making errors that are elementary by human standards.

When Intel sponsored the World Chess Express Challenge in Munich last Friday, they could never have hoped for such a good advertisement for their high-speed Pentium processor. It turned a good computer - Fritz 2 - into a world beater.

In a day of five-minute games, it beat Kasparov, Short, Anand, Kramnik and Gelfand. Finally, Fritz 2 and Kasparov shared first place. Kasparov won the play-off by 4-1, but it is only a matter of time before the machine overtakes even him.

Tomorrow we shall look at Fritz's win against Nigel Short. Meanwhile, here's a sobering thought: Fritz 2 can examine more positions during a single game than a human being can in an entire lifetime.

So Fritz 2 in 1994 was probably the second best player (among humans) in the world in 5 minute blitz. Fritz 5.32 on the CCRL blitz list is 2539 on an i7. Presumably Fritz 2 would be something like 2300-2400 on an i7. But the computers in 1994 were vastly slower, probably also single core. I would guess that the hardware would reduce the CCRL rating by at least another 200 elo. So this puts Kasparov at perhaps 2200 CCRL blitz, the other top humans perhaps 2100 CCRL blitz. I guess this is reasonably consistent with the estimate of 2150 to 2200 for Nakamura on CCRL blitz. So to estimate human blitz ratings for CCRL engines on the blitz list, at least for those in the ballpark of the best humans, we should add 1000 for chess.com blitz or 700 for FIDE blitz. If anyone has data that could make the estimated rating of Fritz 2 on 1994 more accurate, please provide it.