Dragon versus Nakamura

[lkaufman]

I estimated 30 - 100 nodes per move at 15m + 10s TC the range of Nakamura's strength, and for 50 nodes per move I got +3 -0 =5 score for Dragon, which is not much off. I consider that Naka would have gotten a score in this ballpark, had he had more composure and was less distracted. Also, had he prepared better the openings. In terms of nodes per second, 2 nodes per second should match pretty well Naka with this net at a variety of TC. I have chosen Lc0 also because its scaling with TC is more similar to humans than the scaling of AB engines and yes, you are right, we can use the constant NPS speed to simulate Nakamura at variety of TCs. But I do use max NPS Lc0 to actually simulate because it's much faster to simulate, one side (simulated human) plays practically instantly. Also, using some 50 or 70 nodes per move for Nakamura and 10 nodes per move for a much weaker GM comes handily to simulate a variety of GMs, while with nodes per second = 1 one can use only 1, 2, and 3 as values for a human (GM).

Btw, for some reason I am also getting that 2 pawns + move is playable and fair against Nakamura for Dragon MCTS (enabled with a small book), and that Knight odds at 15 + 10 are playable against ~2400-2450 GMs (although Dragon should lose in this case) if the prepared human openings are neutralized (Dragon to have a good small book).

I'm using Ed Schroder's "Benjamin" program as a benchmark to compare Lc0 at low NPS. Benjamin is rated about 2700 CCRL 40/15, and is a logical modern engine to use as it is an improved version of one that played a match with Anand back in 1998 (Ed, if you read this, how much stronger is Benjamin than that 1998 Rebel?). Running on a modern i7 (the reference engine for CCRL), I would expect it to play close to 2800 FIDE at standard time limit, and to be slightly stronger than Nakamura or even Carlsen at Rapid. According to my tests, Lc0 with the j92-330 net needs about 1.6 NPS to break even with "Benjamin" at 15' + 10". So I would estimate that a setting somewhere in the 1 to 1.5 range would best simulate Nakamura in Rapid, and possibly also at other time controls.
Some questions: 1. Why do you say nps values have to be integers, it seems to accept decimal values with no problem?
2. I don't see a place to set nodes per move, only nodes per second, in the 26.3 Lc0 gui. How do you do that?
3. Are you able to give any comparative results for Dragon in standard mode compared to Dragon MCTS in giving handicaps to simulated human?

1. I was probably wrong and one can use decimal and centesimal values like 1.62 nps. I don't use nodes per second as this slows down my experiment by a factor of 2 compared to specifying the nodes per move (and instant move).

2. I don't know what Lc0 GUI is. Something like Nibbler? In Cutechess - Cli or GUI one can specify the number of nodes per move used (Cutechess sends "go nodes" command).

3. I seem to get no better results with Dragon MCTS than with standard Dragon against simulated human as Lc0, unlike of what apparently happened against Nakamura. I am not sure why, maybe MCTS is not that much stronger against Naka as it seemed?

It is very interesting that you came up with 1.6 nps starting with Benjamin plausible human rating, Mark came with 2.3 nps with some extrapolations, and I came with 50 nodes per move or about 2.0 nps starting with our older elaborate estimation of a 1 million nodes of Komodo 10 matching a 2800 GM at 45 + 15. All estimates and extrapolations come amazingly close, not even a factor of 2 off, while unknowns like scaling issues and Elo behavior are very serious.

Sorry, I misspoke about "Lc0 gui". I meant the list of UCI options for Lc0 as shown in the gui I used, the Fritz gui. After reviewing the games, I am inclined to agree with you that the much better engine performance on day 2 was probably due to luck and to Naka going "on tilt" as they say, more than to the MCTS issue. In most key positions both versions chose the same move. At large handicaps MCTS doesn't do well, better to use normal Dragon, but at two pawns MCTS may be a bit better vs. humans.

[Chessqueen]

I estimated 30 - 100 nodes per move at 15m + 10s TC the range of Nakamura's strength, and for 50 nodes per move I got +3 -0 =5 score for Dragon, which is not much off. I consider that Naka would have gotten a score in this ballpark, had he had more composure and was less distracted. Also, had he prepared better the openings. In terms of nodes per second, 2 nodes per second should match pretty well Naka with this net at a variety of TC. I have chosen Lc0 also because its scaling with TC is more similar to humans than the scaling of AB engines and yes, you are right, we can use the constant NPS speed to simulate Nakamura at variety of TCs. But I do use max NPS Lc0 to actually simulate because it's much faster to simulate, one side (simulated human) plays practically instantly. Also, using some 50 or 70 nodes per move for Nakamura and 10 nodes per move for a much weaker GM comes handily to simulate a variety of GMs, while with nodes per second = 1 one can use only 1, 2, and 3 as values for a human (GM).

Btw, for some reason I am also getting that 2 pawns + move is playable and fair against Nakamura for Dragon MCTS (enabled with a small book), and that Knight odds at 15 + 10 are playable against ~2400-2450 GMs (although Dragon should lose in this case) if the prepared human openings are neutralized (Dragon to have a good small book).

I'm using Ed Schroder's "Benjamin" program as a benchmark to compare Lc0 at low NPS. Benjamin is rated about 2700 CCRL 40/15, and is a logical modern engine to use as it is an improved version of one that played a match with Anand back in 1998 (Ed, if you read this, how much stronger is Benjamin than that 1998 Rebel?). Running on a modern i7 (the reference engine for CCRL), I would expect it to play close to 2800 FIDE at standard time limit, and to be slightly stronger than Nakamura or even Carlsen at Rapid. According to my tests, Lc0 with the j92-330 net needs about 1.6 NPS to break even with "Benjamin" at 15' + 10". So I would estimate that a setting somewhere in the 1 to 1.5 range would best simulate Nakamura in Rapid, and possibly also at other time controls.
Some questions: 1. Why do you say nps values have to be integers, it seems to accept decimal values with no problem?
2. I don't see a place to set nodes per move, only nodes per second, in the 26.3 Lc0 gui. How do you do that?
3. Are you able to give any comparative results for Dragon in standard mode compared to Dragon MCTS in giving handicaps to simulated human?

1. I was probably wrong and one can use decimal and centesimal values like 1.62 nps. I don't use nodes per second as this slows down my experiment by a factor of 2 compared to specifying the nodes per move (and instant move).

2. I don't know what Lc0 GUI is. Something like Nibbler? In Cutechess - Cli or GUI one can specify the number of nodes per move used (Cutechess sends "go nodes" command).

3. I seem to get no better results with Dragon MCTS than with standard Dragon against simulated human as Lc0, unlike of what apparently happened against Nakamura. I am not sure why, maybe MCTS is not that much stronger against Naka as it seemed?

It is very interesting that you came up with 1.6 nps starting with Benjamin plausible human rating, Mark came with 2.3 nps with some extrapolations, and I came with 50 nodes per move or about 2.0 nps starting with our older elaborate estimation of a 1 million nodes of Komodo 10 matching a 2800 GM at 45 + 15. All estimates and extrapolations come amazingly close, not even a factor of 2 off, while unknowns like scaling issues and Elo behavior are very serious.

Sorry, I misspoke about "Lc0 gui". I meant the list of UCI options for Lc0 as shown in the gui I used, the Fritz gui. After reviewing the games, I am inclined to agree with you that the much better engine performance on day 2 was probably due to luck and to Naka going "on tilt" as they say, more than to the MCTS issue. In most key positions both versions chose the same move. At large handicaps MCTS doesn't do well, better to use normal Dragon, but at two pawns MCTS may be a bit better vs. humans.

Note: I have been watching the Skilling Open and Noticed that in most 4 or 5 men endgames these super GMs do NOT make the best moves according to engines that are analyzing it, including Carlsen has made several inaccurate moves, so my question here is why do Engines need 6 and 7 men Endgames when playing versus any Super GM, especially in Rapid or Blitz games like that match Dragon MCTS Vs Nakamura?

Even a simple endgane where Naka was playing versus the Indian grandmaster in skilling, Naka played like 50 moves without realizing this simple Lucerna endgame which is DRAW [d]2B2R2/8/8/8/5Krk/8/8/8 w - - 3 1

Note: What would be a fair time control Naka Vs Dragon with 2 pawns Odds 20 minutes and 10 sec increment or 30 minutes and 10 sec increment ?