CCRL Chess Engine Match Standards. How obsolete are they?

[mkchan]

To me this seems more of an attack directed at CCRL team than actually getting to the point. From what I read, the issue is the advertised 40/4 | 40/40 which are actually scaled to some CPU that was decided when the website was started. They clearly state, right at the start, approximates for modern CPUs and the benchmarking methodology. I see no bait and switch here at all. The rating list is still indicative of relative strengths of engines to a pretty good accuracy.

If you have such valuable criticism to make, why not start a list of your own instead of making the entire established community conform to your personal interpretation of their list. In fact, make it pay-to-view as well because it's going to be the one true list with exact rating values measured for each new CPU that comes out right? Don't forget to get a few GMs into the pool to better reflect FIDE rating numbers so that we're not fooling anyone about SF's 3400 rating. I'm sure everyone would flock to that

I was told by subject matter expert that CCRL standards is very poor.

And it is clear reading the posts of the discussion. The practice of CCRL claiming the ratings test as 4/40 and 40/40 needs to be addressed. CCRL can fix this issue today.

So valuable criticism has been made.

No, you have not made valid criticism at all. Why don't you address the point I made in their defense: CCRL shows accurate relative difference between engine strength. You do realize changing time control will not change that by much? Rating is relative to the pool of opponents anyways

[mwyoung]

To me this seems more of an attack directed at CCRL team than actually getting to the point. From what I read, the issue is the advertised 40/4 | 40/40 which are actually scaled to some CPU that was decided when the website was started. They clearly state, right at the start, approximates for modern CPUs and the benchmarking methodology. I see no bait and switch here at all. The rating list is still indicative of relative strengths of engines to a pretty good accuracy.

If you have such valuable criticism to make, why not start a list of your own instead of making the entire established community conform to your personal interpretation of their list. In fact, make it pay-to-view as well because it's going to be the one true list with exact rating values measured for each new CPU that comes out right? Don't forget to get a few GMs into the pool to better reflect FIDE rating numbers so that we're not fooling anyone about SF's 3400 rating. I'm sure everyone would flock to that

I was told by subject matter expert that CCRL standards is very poor.

And it is clear reading the posts of the discussion. The practice of CCRL claiming the ratings test as 4/40 and 40/40 needs to be addressed. CCRL can fix this issue today.

So valuable criticism has been made.

No, you have not made valid criticism at all. Why don't you address the point I made in their defense: CCRL shows accurate relative difference between engine strength. You do realize changing time control will not change that by much? Rating is relative to the pool of opponents anyways

Are you kidding, CCRL can not even rank the best engine. The last time I check on modern hardware. Lc0 was the strongest engine.
But CCRL is blind to this because they test on a benchmark of a CPU from the year 2005. This is the year 2019....


Rank Name Rating Score Average
Opponent Draws Games LOS
Elo + −
1 Stockfish 10 64-bit 4CPU 3546 +13 −12 69.6% −124.9 54.9% 2015
100.0%
2 Houdini 6 64-bit 4CPU 3519 +9 −9 65.5% −108.4 53.9% 3912
95.8%
3 Komodo 11.2 64-bit 4CPU 3503 +16 −16 58.2% −66.6 55.3% 1158
90.4%
4 Lc0 0.21.1 JH.T6.532 GPU 3487 +17 −17 59.2% −58.5 52.4% 1100

[AndrewGrant]

Mike is this better than a CCRL Smart phone? My computer works just fine. What are you talking about....

viewtopic.php?p=773654#p773654

Do yourself a favour and read through that. Its the story of how some people failed to listen about what pondering and hyperthreading can do. Ironically, you've actually managed to take it one step further by assigning more threads than their are hyperthreads, but thats another story.

In case you are too stubborn or lazy to actually read the above link and humour the possibility that you are wrong, I'll give you a summary.

1) You are allocating more threads than there are on the CPU
2) This IS causing engines to be subtly screwed out of CPU time. Ironically, its likely that Leela is hurt more than the AB engines
3) Even without over allocation, there is no way to trust the OS to not lock an engine out of a thread for a period of time.

You are very ignorant on how modern AMD SMT works on a 2950x. Or maybe you are confusing Lc0 for a AB engine. Maybe I am using magic beans.

My computer performance speaks for itself and this is streaming live video with this performance.

Please explain after observing your ignorance....

All games are recorded. And can be viewed. Ponder matches and non Ponder matches. Modern technology is amazing. You need to keep up to date. Never mind I forgot you are stuck in 2005 with CCRL.

Live Link: https://www.youtube.com/watch?v=QKUPqFE-3yw

Send me the full UCI logs from any of your matches and I will find hundreds of issues.

If you TOOK the time to read that thread, you would understand this. I am going to follow gunther's advice, and save myself the trouble of reading your postings.

[mwyoung]

Mike is this better than a CCRL Smart phone? My computer works just fine. What are you talking about....

viewtopic.php?p=773654#p773654

Do yourself a favour and read through that. Its the story of how some people failed to listen about what pondering and hyperthreading can do. Ironically, you've actually managed to take it one step further by assigning more threads than their are hyperthreads, but thats another story.

In case you are too stubborn or lazy to actually read the above link and humour the possibility that you are wrong, I'll give you a summary.

1) You are allocating more threads than there are on the CPU
2) This IS causing engines to be subtly screwed out of CPU time. Ironically, its likely that Leela is hurt more than the AB engines
3) Even without over allocation, there is no way to trust the OS to not lock an engine out of a thread for a period of time.

You are very ignorant on how modern AMD SMT works on a 2950x. Or maybe you are confusing Lc0 for a AB engine. Maybe I am using magic beans.


My computer performance speaks for itself and this is streaming live video with this performance.

Please explain after observing your ignorance....

All games are recorded. And can be viewed. Ponder matches and non Ponder matches. Modern technology is amazing. You need to keep up to date. Never mind I forgot you are stuck in 2005 with CCRL.

Live Link: https://www.youtube.com/watch?v=QKUPqFE-3yw

Send me the full UCI logs from any of your matches and I will find hundreds of issues.

If you TOOK the time to read that thread, you would understand this. I am going to follow gunther's advice, and save myself the trouble of reading your postings.

You can't explain it mister subject matter expert.
I think you should go with the magic bean theory.

holy-grail-run-away.jpg

[mkchan]

To me this seems more of an attack directed at CCRL team than actually getting to the point. From what I read, the issue is the advertised 40/4 | 40/40 which are actually scaled to some CPU that was decided when the website was started. They clearly state, right at the start, approximates for modern CPUs and the benchmarking methodology. I see no bait and switch here at all. The rating list is still indicative of relative strengths of engines to a pretty good accuracy.

If you have such valuable criticism to make, why not start a list of your own instead of making the entire established community conform to your personal interpretation of their list. In fact, make it pay-to-view as well because it's going to be the one true list with exact rating values measured for each new CPU that comes out right? Don't forget to get a few GMs into the pool to better reflect FIDE rating numbers so that we're not fooling anyone about SF's 3400 rating. I'm sure everyone would flock to that

I was told by subject matter expert that CCRL standards is very poor.

And it is clear reading the posts of the discussion. The practice of CCRL claiming the ratings test as 4/40 and 40/40 needs to be addressed. CCRL can fix this issue today.

So valuable criticism has been made.

No, you have not made valid criticism at all. Why don't you address the point I made in their defense: CCRL shows accurate relative difference between engine strength. You do realize changing time control will not change that by much? Rating is relative to the pool of opponents anyways

Are you kidding, CCRL can not even rank the best engine. The last time I check on modern hardware. Lc0 was the strongest engine.
But CCRL is blind to this because they test on a benchmark of a CPU from the year 2005. This is the year 2019....


Rank Name Rating Score Average
Opponent Draws Games LOS
Elo + −
1 Stockfish 10 64-bit 4CPU 3546 +13 −12 69.6% −124.9 54.9% 2015
100.0%
2 Houdini 6 64-bit 4CPU 3519 +9 −9 65.5% −108.4 53.9% 3912
95.8%
3 Komodo 11.2 64-bit 4CPU 3503 +16 −16 58.2% −66.6 55.3% 1158
90.4%
4 Lc0 0.21.1 JH.T6.532 GPU 3487 +17 −17 59.2% −58.5 52.4% 1100

What? Lc0 might be best on an RTX 2080 sure but not everyone is going to be able to afford that. In either case, they clearly state they tested on Gtx 1050 with the ratio mentioned. I don't know what kind of a fool you take the people of this forum for but that is, clearly, the rating given the hardware. Sure if you scale up the hardware the rating changes (oh I wonder if CCRL has mentioned that scaling hmm..... Oh look! It's on the front page and the lc0 page).
Like I said, you can go ahead and make your rating list with the latest "modern" hardware to satisfy yourself. CCRL according EVERYONE else is fine

[mwyoung]

To me this seems more of an attack directed at CCRL team than actually getting to the point. From what I read, the issue is the advertised 40/4 | 40/40 which are actually scaled to some CPU that was decided when the website was started. They clearly state, right at the start, approximates for modern CPUs and the benchmarking methodology. I see no bait and switch here at all. The rating list is still indicative of relative strengths of engines to a pretty good accuracy.

If you have such valuable criticism to make, why not start a list of your own instead of making the entire established community conform to your personal interpretation of their list. In fact, make it pay-to-view as well because it's going to be the one true list with exact rating values measured for each new CPU that comes out right? Don't forget to get a few GMs into the pool to better reflect FIDE rating numbers so that we're not fooling anyone about SF's 3400 rating. I'm sure everyone would flock to that

I was told by subject matter expert that CCRL standards is very poor.

And it is clear reading the posts of the discussion. The practice of CCRL claiming the ratings test as 4/40 and 40/40 needs to be addressed. CCRL can fix this issue today.

So valuable criticism has been made.

No, you have not made valid criticism at all. Why don't you address the point I made in their defense: CCRL shows accurate relative difference between engine strength. You do realize changing time control will not change that by much? Rating is relative to the pool of opponents anyways

Are you kidding, CCRL can not even rank the best engine. The last time I check on modern hardware. Lc0 was the strongest engine.
But CCRL is blind to this because they test on a benchmark of a CPU from the year 2005. This is the year 2019....


Rank Name Rating Score Average
Opponent Draws Games LOS
Elo + −
1 Stockfish 10 64-bit 4CPU 3546 +13 −12 69.6% −124.9 54.9% 2015
100.0%
2 Houdini 6 64-bit 4CPU 3519 +9 −9 65.5% −108.4 53.9% 3912
95.8%
3 Komodo 11.2 64-bit 4CPU 3503 +16 −16 58.2% −66.6 55.3% 1158
90.4%
4 Lc0 0.21.1 JH.T6.532 GPU 3487 +17 −17 59.2% −58.5 52.4% 1100

What? Lc0 might be best on an RTX 2080 sure but not everyone is going to be able to afford that. In either case, they clearly state they tested on Gtx 1050 with the ratio mentioned. I don't know what kind of a fool you take the people of this forum for but that is, clearly, the rating given the hardware. Sure if you scale up the hardware the rating changes (oh I wonder if CCRL has mentioned that scaling hmm..... Oh look! It's on the front page and the lc0 page).
Like I said, you can go ahead and make your rating list with the latest "modern" hardware to satisfy yourself. CCRL according EVERYONE else is fine

It is not fine. That is why more and more people look to TCEC for what engine is best. Why, Because we live in the year 2019. When CCRL is still trying to test with 2005 hardware. Facts can hurt....

[xr_a_y]

It is not fine. That is why more and more people look to TCEC for what engine is best. Why, Because we live in the year 2019. When CCRL is still trying to test with 2005 hardware. Facts can hurt....

Given the very small number of games played in TCEC, the given ratings are very blury (in term of elo it shall be +/- 200), hard to say which engine is the best ...

To "order" engines, you need at least 300 or 1000 games if elo difference between them is greater than 15. To get smaller elo gap you'll need 10000 games or even more.

Let me ask a question again : what would be the point of testing engine on nowadays hardware at 40/40min ? (versus 40/10 or 40/15 as it is the case today). What are you expecting from such test ?

[mwyoung]

It is not fine. That is why more and more people look to TCEC for what engine is best. Why, Because we live in the year 2019. When CCRL is still trying to test with 2005 hardware. Facts can hurt....

Given the very small number of games played in TCEC, the given ratings are very blury (in term of elo it shall be +/- 200), hard to say which engine is the best ...

To "order" engines, you need at least 300 or 1000 games if elo difference between them is greater than 15. To get smaller elo gap you'll need 10000 games or even more.

Let me ask a question again : what would be the point of testing engine on nowadays hardware at 40/40min ? (versus 40/10 or 40/15 as it is the case today). What are you expecting from such test ?

I agree, but we are computer chess geeks. I know because I live in St. Louis. And the St. Louis chess club has many GM players and ranked players. They are really very uninformed on computer chess. They now of TCEC and who wins, and the other players follow their lead. This is what most of them know when I talk with them. Not all players are GM Larry K. who knows both chess and computer chess.

I will say GM Alexander Shabalov impressed me with his computer chess knowledge, but he was the exception.

If you want fast time controls then run real matches at 40/10. Using modern hardware. I do expect a standard better then a year 2005 cpu running on only 1 or 4 threads.

I do expect honesty in the testing. Regardless of hardware. SSDF and other don't mislead reader.


The Swedish Ratinglist may be quoted in other magazines, but we insist that this will be done in a correct way! We expect, that not only the rating figures, but also the number of games and the margin of error will be quoted.

Please read the comment by the chairman, Lars Sandin. You may also download the list in DOS text format. Please note that this is a longer list, with almost all tested computers since SSDF began its work more than 20 years ago!

All games have been played on the tournament level, 40 moves/2 hours followed by 20 moves/each following hour. In matches between PC-programs, two separate PCs have been used, connected with an auto232-cable.

If you have any questions about the list you are welcome to contact us.



Rating + - Games Won Av.opp
1 Stockfish 9 x64 1800X 3.6 GHz 3494 32 -30 642 74% 3308
2 Komodo 12.3 x64 1800X 3.6 GHz 3456 30 -28 640 68% 3321
3 Stockfish 9 x64 Q6600 2.4 GHz 3446 50 -48 200 57% 3396
4 Stockfish 8 x64 1800X 3.6 GHz 3432 26 -24 1059 77% 3217
5 Stockfish 8 x64 Q6600 2.4 GHz 3418 38 -35 440 72% 3251
6 Komodo 11.01 x64 1800X 3.6 GHz 3397 23 -22 1134 72% 3229
7 Deep Shredder 13 x64 1800X 3.6 GHz 3360 25 -24 830 66% 3246
8 Booot 6.3.1 x64 1800X 3.6 GHz 3352 29 -29 560 54% 3319
9 Komodo 9.1 x64 Q6600 2.4 GHz 3340 21 -20 1435

[mkchan]

To me this seems more of an attack directed at CCRL team than actually getting to the point. From what I read, the issue is the advertised 40/4 | 40/40 which are actually scaled to some CPU that was decided when the website was started. They clearly state, right at the start, approximates for modern CPUs and the benchmarking methodology. I see no bait and switch here at all. The rating list is still indicative of relative strengths of engines to a pretty good accuracy.

If you have such valuable criticism to make, why not start a list of your own instead of making the entire established community conform to your personal interpretation of their list. In fact, make it pay-to-view as well because it's going to be the one true list with exact rating values measured for each new CPU that comes out right? Don't forget to get a few GMs into the pool to better reflect FIDE rating numbers so that we're not fooling anyone about SF's 3400 rating. I'm sure everyone would flock to that

I was told by subject matter expert that CCRL standards is very poor.

And it is clear reading the posts of the discussion. The practice of CCRL claiming the ratings test as 4/40 and 40/40 needs to be addressed. CCRL can fix this issue today.

So valuable criticism has been made.

No, you have not made valid criticism at all. Why don't you address the point I made in their defense: CCRL shows accurate relative difference between engine strength. You do realize changing time control will not change that by much? Rating is relative to the pool of opponents anyways

Are you kidding, CCRL can not even rank the best engine. The last time I check on modern hardware. Lc0 was the strongest engine.
But CCRL is blind to this because they test on a benchmark of a CPU from the year 2005. This is the year 2019....


Rank Name Rating Score Average
Opponent Draws Games LOS
Elo + −
1 Stockfish 10 64-bit 4CPU 3546 +13 −12 69.6% −124.9 54.9% 2015
100.0%
2 Houdini 6 64-bit 4CPU 3519 +9 −9 65.5% −108.4 53.9% 3912
95.8%
3 Komodo 11.2 64-bit 4CPU 3503 +16 −16 58.2% −66.6 55.3% 1158
90.4%
4 Lc0 0.21.1 JH.T6.532 GPU 3487 +17 −17 59.2% −58.5 52.4% 1100

What? Lc0 might be best on an RTX 2080 sure but not everyone is going to be able to afford that. In either case, they clearly state they tested on Gtx 1050 with the ratio mentioned. I don't know what kind of a fool you take the people of this forum for but that is, clearly, the rating given the hardware. Sure if you scale up the hardware the rating changes (oh I wonder if CCRL has mentioned that scaling hmm..... Oh look! It's on the front page and the lc0 page).
Like I said, you can go ahead and make your rating list with the latest "modern" hardware to satisfy yourself. CCRL according EVERYONE else is fine

It is not fine. That is why more and more people look to TCEC for what engine is best. Why, Because we live in the year 2019. When CCRL is still trying to test with 2005 hardware. Facts can hurt....

No one who knows anything about testing looks at TCEC for what engine is best. It is a tournament, not a dedicated effort into testing. You are simply appealing to authority and not contradicting the actual point I'm making with facts:

>CCRL shows accurate relative difference between engine strength. You do realize changing time control will not change that by much? Rating is relative to the pool of opponents anyways. Regardless of what you or the masses think, mathematically CCRL > TCEC for rating estimates

[mwyoung]

To me this seems more of an attack directed at CCRL team than actually getting to the point. From what I read, the issue is the advertised 40/4 | 40/40 which are actually scaled to some CPU that was decided when the website was started. They clearly state, right at the start, approximates for modern CPUs and the benchmarking methodology. I see no bait and switch here at all. The rating list is still indicative of relative strengths of engines to a pretty good accuracy.

If you have such valuable criticism to make, why not start a list of your own instead of making the entire established community conform to your personal interpretation of their list. In fact, make it pay-to-view as well because it's going to be the one true list with exact rating values measured for each new CPU that comes out right? Don't forget to get a few GMs into the pool to better reflect FIDE rating numbers so that we're not fooling anyone about SF's 3400 rating. I'm sure everyone would flock to that

I was told by subject matter expert that CCRL standards is very poor.

And it is clear reading the posts of the discussion. The practice of CCRL claiming the ratings test as 4/40 and 40/40 needs to be addressed. CCRL can fix this issue today.

So valuable criticism has been made.

No, you have not made valid criticism at all. Why don't you address the point I made in their defense: CCRL shows accurate relative difference between engine strength. You do realize changing time control will not change that by much? Rating is relative to the pool of opponents anyways

Are you kidding, CCRL can not even rank the best engine. The last time I check on modern hardware. Lc0 was the strongest engine.
But CCRL is blind to this because they test on a benchmark of a CPU from the year 2005. This is the year 2019....


Rank Name Rating Score Average
Opponent Draws Games LOS
Elo + −
1 Stockfish 10 64-bit 4CPU 3546 +13 −12 69.6% −124.9 54.9% 2015
100.0%
2 Houdini 6 64-bit 4CPU 3519 +9 −9 65.5% −108.4 53.9% 3912
95.8%
3 Komodo 11.2 64-bit 4CPU 3503 +16 −16 58.2% −66.6 55.3% 1158
90.4%
4 Lc0 0.21.1 JH.T6.532 GPU 3487 +17 −17 59.2% −58.5 52.4% 1100

What? Lc0 might be best on an RTX 2080 sure but not everyone is going to be able to afford that. In either case, they clearly state they tested on Gtx 1050 with the ratio mentioned. I don't know what kind of a fool you take the people of this forum for but that is, clearly, the rating given the hardware. Sure if you scale up the hardware the rating changes (oh I wonder if CCRL has mentioned that scaling hmm..... Oh look! It's on the front page and the lc0 page).
Like I said, you can go ahead and make your rating list with the latest "modern" hardware to satisfy yourself. CCRL according EVERYONE else is fine

It is not fine. That is why more and more people look to TCEC for what engine is best. Why, Because we live in the year 2019. When CCRL is still trying to test with 2005 hardware. Facts can hurt....

No one who knows anything about testing looks at TCEC for what engine is best. It is a tournament, not a dedicated effort into testing. You are simply appealing to authority and not contradicting the actual point I'm making with facts:

>CCRL shows accurate relative difference between engine strength. You do realize changing time control will not change that by much? Rating is relative to the pool of opponents anyways. Regardless of what you or the masses think, mathematically CCRL > TCEC for rating estimates

The world of chess engines is bigger then us here. Most people who use chess engines knows nothing of CCC. Or are they in the weeds about computer chess testing.

I live in both worlds.