Next Komodo vs. GM handicap match - "Standard chess&quo

[Ozymandias]

Considering all of the calculations below in this thread and the seeming agreement between those involved, I would find it much more interesting if Komodo's time were set to 1'+0.3" or 45"+0.25". Otherwise this is yet another match where the engine is relatively overpowered compared to the human.

If we see a more balanced match it is also somewhat easier to estimate the true strength of the computer at this particular speed versus a match that ends 4 to zero.

Already addressed:

Aside from the unsettling aspect of an instant move, I don't think many Komodo owners care how well Komodo plays in 0.1" for example. Anything much faster than 3' + 1" is just not relevant to human users

[mbabigian]

If these events are to simulate how owners use the software, then they fail on their face. In fact, every rating list published fails to represent how I typically use the software too.

If however, we are trying to see approximately how many ELO beyond humans, the engines have gotten, a test with an even score is best.

Again, just my 2 cents.
Mike.

[lkaufman]

If these events are to simulate how owners use the software, then they fail on their face. In fact, every rating list published fails to represent how I typically use the software too.

If however, we are trying to see approximately how many ELO beyond humans, the engines have gotten, a test with an even score is best.

Again, just my 2 cents.
Mike.

So how do you use the software, and what test would best simulate it?

Playing a 2600 GM on totally even terms would probably produce a 100% score or close to it, which would tell us nothing about Komodo's elo other than that it starts with a "3".

[Jesse Gersenson]

FIDE 2591 ... 1.36 million nps in the opening position.

Komodo gave two pawns and won against a FIDE 2520 running on a machine which got 474,000 nps in the opening position after 60 seconds.

I predict the GM will be stomped in at least 3.5 of the games.

Average depth of 21-22? This'll be brutal.

[Ozymandias]

running on a machine which got 474,000 ops in the opening position after 60 seconds

Was it this one?

[mbabigian]

Larry, I use chess software for analysis and nearly every chess player I know does also. Almost no one I know actually plays the software and zero, absolutely zero, chess players in my local community play engine vs engine games. It is very easy to read CCC and other forums that revolve around computer chess and get the misconception that the vast majority of chess players sit around pitting their fast hardware against other computers.

That being said, when I am investigating a chess position, I would find anything less than 60 seconds of thinking to be poor analysis "relatively speaking." I rarely run analysis up to correspondence time frames, but using "modern hardware" 2 minutes to a few hours is the typical range for analysis. That being said, it is not practical for you to construct a match to simulate this usage.

Second, your comment that in an even match the score would be a 100% misses my point. I did not suggest a match ending in a 100% score. I suggested a match were the end score is split 50/50. The task I would give you is finding the conditions that would create that 50/50 split. If it was done with time handicapping only, it would be easy to conservatively project the engine's ELO to tournament time-frames via formula. I would be curious to know where that ELO lies. As it stands, all we really know is that computer vs computer match generated ELOs are not indicative of human ratings and that engines play North of the best humans.

I hope that makes sense,
Mike

[lkaufman]

So the current test is rather relevant for you, since the conditions are expected to bring the engine level down below the top human player, though probably not down to the chosen opponent. We don't really know though, so it is certainly possible that he will make an even score.

[mbabigian]

Yes, I very much hope it is at least a relatively close contest. If it is, I think we could learn a bit more about the current relative level of play between the two species. We'd just need more of these contests to accumulate a base of games.

I'm certainly not hoping for a 3.5/0.5 result

All that said, I find all of your human/Komodo matches entertaining, and I don't want you to think I'm just complaining about your efforts. They are appreciated! I am just curious if we could better extrapolate where modern hardware at tournament time controls sit from your handicap matches. I think we can, if we do what is necessary to level the playing field and then adjust the rating to say a typical 6way with 3min/move.

Again, just my thoughts/wishes,
Mike

[lkaufman]

Yes, I very much hope it is at least a relatively close contest. If it is, I think we could learn a bit more about the current relative level of play between the two species. We'd just need more of these contests to accumulate a base of games.

I'm certainly not hoping for a 3.5/0.5 result

All that said, I find all of your human/Komodo matches entertaining, and I don't want you to think I'm just complaining about your efforts. They are appreciated! I am just curious if we could better extrapolate where modern hardware at tournament time controls sit from your handicap matches. I think we can, if we do what is necessary to level the playing field and then adjust the rating to say a typical 6way with 3min/move.

Again, just my thoughts/wishes,
Mike

The problem is that although we have pretty good data on how much doubling speed or time is worth in engine vs engine play, we can only guess how this translates to elo points against humans. But we have some idea.

[mhull]

Yes, I very much hope it is at least a relatively close contest. If it is, I think we could learn a bit more about the current relative level of play between the two species. We'd just need more of these contests to accumulate a base of games.

I'm certainly not hoping for a 3.5/0.5 result

All that said, I find all of your human/Komodo matches entertaining, and I don't want you to think I'm just complaining about your efforts. They are appreciated! I am just curious if we could better extrapolate where modern hardware at tournament time controls sit from your handicap matches. I think we can, if we do what is necessary to level the playing field and then adjust the rating to say a typical 6way with 3min/move.

Again, just my thoughts/wishes,
Mike

The problem is that although we have pretty good data on how much doubling speed or time is worth in engine vs engine play, we can only guess how this translates to elo points against humans. But we have some idea.

I'm curios about a notional metric I like to call "path-length-to-Elo" or PLTE. Or at what processor speed must a given program run to play at some arbitrary human strength, say 2200, 2400 or 2600? Each program probably has a unique PLTE or PLTE curve.

Questions related to this metric might be:

1) Knowing a program's PLTE curve and setting a processor speed for it to play at a human ELO of 2500, could grandmaster sparring at this level better contribute to further improving PLTE as opposed to self-play or sparring with other programs at full speed?

2) How much has this metric changed over time with the introduction of software innovations. For example, with current knowledge, could we necessarily decrease PLTE on legacy hardware (e.g. MC68000, ARM, 6502) used by period programs?

3) Or has stronger play required an overall higher PLTE at a some Elo threshold?

4) Does anyone else think this is interesting?