TalkChess.com

I'd be curious to know where Komodo 13 lvl 24 or Komodo 11 lvl 19 would be located in the list ccrl 40/40 and ccrl 40/4, since their strength is absolute and not relative.
Has anyone done similar tests?

Marcus9 wrote: ↑Mon Aug 19, 2019 12:33 am I'd be curious to know where Komodo 13 lvl 24 or Komodo 11 lvl 19 would be located in the list ccrl 40/40 and ccrl 40/4, since their strength is absolute and not relative.
Has anyone done similar tests?

Same from me, + Stockfish level 19 and level 18.

Marcus9 wrote: ↑Mon Aug 19, 2019 12:33 am I'd be curious to know where Komodo 13 lvl 24 or Komodo 11 lvl 19 would be located in the list ccrl 40/40 and ccrl 40/4, since their strength is absolute and not relative.
Has anyone done similar tests?

When Komodo 13.1 is released, each level up to 22 will correspond to 3 levels lower in older Komodo versions. So old level 0 is new level 3, old level 19 is new level 22. New levels 1 and 2 are weaker than old level 0, and new levels 23 and 24 are stronger than old level 19 (but weaker than old level 20, which is full strength, now level 25). We have lots of data as to the strength of these levels against humans at "slow blitz" levels on chess.com, but not much data vs engines. One problem is that I believe that CCRL doesn't really run the 40/4 games at the "adapted" time controls but uses a minimum of 40/3 min regardless of hardware (I may be wrong, perhaps it is CEGT that does this, or perhaps both of them). This won't matter much for engines that actually use the time, but it would affect the rating of the Komodo levels a bit, they would get higher ratings on slower hardware. But probably not a big deal. I could run a test on our tester at say 3' + 1" to get some estimate of the blitz rating of new level 24. I'm pretty sure that new level 24 would win most of the games at blitz (or even "slow blitz" like 5' + 5") against any human (Naka, Carlsen, or MVL), since very few GMs other than Naka and MVL were able to beat what is now level 22 in blitz.

Vinvin wrote: ↑Mon Aug 19, 2019 2:51 am

Marcus9 wrote: ↑Mon Aug 19, 2019 12:33 am I'd be curious to know where Komodo 13 lvl 24 or Komodo 11 lvl 19 would be located in the list ccrl 40/40 and ccrl 40/4, since their strength is absolute and not relative.
Has anyone done similar tests?

Same from me, + Stockfish level 19 and level 18.

OK, I ran some tests at 3' + 1" on our tester, and while the results seemed to vary more than I would expect from opponent to opponent, in round numbers the ccrl 40/4 rating for Komodo 13.1 level 24 should be in the ballpark of 2800. Note that the CCRL 40/4 ratings are much lower than what these engines would get on the FIDE blitz list if they played with humans, maybe 400 elo lower. So this 2800 probably means a FIDE blitz of 3200 or so. As for CCRL 40/40 since the levels don't use the extra time they would drop by something like 250 to 300 elo. The levels below 24 are supposed to be spaced at about 150 elo per level on engine lists or 125 elo per level in human elo terms. So levels 1 thru 3 should actually be below 0 on the CCRL scale.

lkaufman wrote: ↑Mon Aug 19, 2019 5:00 am

Vinvin wrote: ↑Mon Aug 19, 2019 2:51 am

Marcus9 wrote: ↑Mon Aug 19, 2019 12:33 am I'd be curious to know where Komodo 13 lvl 24 or Komodo 11 lvl 19 would be located in the list ccrl 40/40 and ccrl 40/4, since their strength is absolute and not relative.
Has anyone done similar tests?

Same from me, + Stockfish level 19 and level 18.

OK, I ran some tests at 3' + 1" on our tester, and while the results seemed to vary more than I would expect from opponent to opponent, in round numbers the ccrl 40/4 rating for Komodo 13.1 level 24 should be in the ballpark of 2800. Note that the CCRL 40/4 ratings are much lower than what these engines would get on the FIDE blitz list if they played with humans, maybe 400 elo lower. So this 2800 probably means a FIDE blitz of 3200 or so. As for CCRL 40/40 since the levels don't use the extra time they would drop by something like 250 to 300 elo. The levels below 24 are supposed to be spaced at about 150 elo per level on engine lists or 125 elo per level in human elo terms. So levels 1 thru 3 should actually be below 0 on the CCRL scale.

unfortunately the CCRL test many buggy engines so the random player has more than 0 elo in the CCRL list.

Some weak engines simply draw often by repetition or by stalemate against the random mover when they have a huge material advantage.
I think it is better simply not to allow testing them and it is not logical that a player that does not give material by one ply blunders is going to do these types of blunders but I am not responsible for the CCRL list.

Uri Blass wrote: ↑Mon Aug 19, 2019 6:26 am

lkaufman wrote: ↑Mon Aug 19, 2019 5:00 am

Vinvin wrote: ↑Mon Aug 19, 2019 2:51 am

Marcus9 wrote: ↑Mon Aug 19, 2019 12:33 am I'd be curious to know where Komodo 13 lvl 24 or Komodo 11 lvl 19 would be located in the list ccrl 40/40 and ccrl 40/4, since their strength is absolute and not relative.
Has anyone done similar tests?

Same from me, + Stockfish level 19 and level 18.

OK, I ran some tests at 3' + 1" on our tester, and while the results seemed to vary more than I would expect from opponent to opponent, in round numbers the ccrl 40/4 rating for Komodo 13.1 level 24 should be in the ballpark of 2800. Note that the CCRL 40/4 ratings are much lower than what these engines would get on the FIDE blitz list if they played with humans, maybe 400 elo lower. So this 2800 probably means a FIDE blitz of 3200 or so. As for CCRL 40/40 since the levels don't use the extra time they would drop by something like 250 to 300 elo. The levels below 24 are supposed to be spaced at about 150 elo per level on engine lists or 125 elo per level in human elo terms. So levels 1 thru 3 should actually be below 0 on the CCRL scale.

unfortunately the CCRL test many buggy engines so the random player has more than 0 elo in the CCRL list.

Some weak engines simply draw often by repetition or by stalemate against the random mover when they have a huge material advantage.
I think it is better simply not to allow testing them and it is not logical that a player that does not give material by one ply blunders is going to do these types of blunders but I am not responsible for the CCRL list.

I forgot that the CCRL ratings (unlike CEGT) are contracted by the use of BayesElo. My 150 per level estimate was for Ordo (used by CEGT); the 125 figure should be about right for CCRL/BayesElo. So I revise my estimates for the low levels to -75 for level 1, +50 for level 2, and +175 for level 3. Still these sound too low in view of your comment about random player elo. I suppose it just means we can't take these low ratings seriously.

lkaufman wrote: ↑Mon Aug 19, 2019 3:27 am One problem is that I believe that CCRL doesn't really run the 40/4 games at the "adapted" time controls but uses a minimum of 40/3 min regardless of hardware (I may be wrong, perhaps it is CEGT that does this, or perhaps both of them).

Yes you are wrong in respect of CCRL. Many CCRL blitz games these days are played at an adapted 40/2 time control.

Modern Times wrote: ↑Mon Aug 19, 2019 7:21 am

lkaufman wrote: ↑Mon Aug 19, 2019 3:27 am One problem is that I believe that CCRL doesn't really run the 40/4 games at the "adapted" time controls but uses a minimum of 40/3 min regardless of hardware (I may be wrong, perhaps it is CEGT that does this, or perhaps both of them).

Yes you are wrong in respect of CCRL. Many CCRL blitz games these days are played at an adapted 40/2 time control.

Thanks. Just for clarity, are you saying that all CCRL testers are supposed to use whatever tc is properly adapted for their hardware, or do they have the option of using 40/3 even if it should be 40/2 (for example)? Or is it just a question of which way to round to the nearest minute?

lkaufman wrote: ↑Mon Aug 19, 2019 7:52 am Thanks. Just for clarity, are you saying that all CCRL testers are supposed to use whatever tc is properly adapted for their hardware,

Yes.

lkaufman wrote: ↑Mon Aug 19, 2019 7:52 am Or is it just a question of which way to round to the nearest minute?

Some GUIs don't accept minutes and seconds so you have to round. Others accept minutes and seconds.