CCRL Chess Engine Match Standards. How obsolete are they?

Discussion of computer chess matches and engine tournaments.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
mwyoung
Posts: 1641
Joined: Wed May 12, 2010 8:00 pm

CCRL Chess Engine Match Standards. How obsolete are they?

Post by mwyoung » Sun Jun 30, 2019 2:32 pm

CCRL Chess Engine Testing Standards..jpg
CCRL Chess Engine Testing Standards..jpg (228.54 KiB) Viewed 2188 times
Professing themselves to be wise, they became fools,
Take on me. foes 0

elcabesa
Posts: 815
Joined: Sun May 23, 2010 11:32 am
Contact:

Re: CCRL Chess Engine Match Standards. How obsolete are they?

Post by elcabesa » Sun Jun 30, 2019 3:43 pm

I think it mainly means "consistency"

User avatar
xr_a_y
Posts: 764
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: CCRL Chess Engine Match Standards. How obsolete are they?

Post by xr_a_y » Sun Jun 30, 2019 4:07 pm

So what ?, 40/40 means something like 40/10 and 40/4 means something like 40/1. This is not an issue I think, CCRL testers are using heterogeneous hardware anyway, so "scaling" is alwas needed.

Do you really want engines to be tested on 8 threads @3.8GHz, 32Gb hash for 40 min TC ? For what purpose ?

AndrewGrant
Posts: 490
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: CCRL Chess Engine Match Standards. How obsolete are they?

Post by AndrewGrant » Sun Jun 30, 2019 4:26 pm

mwyoung wrote:
Sun Jun 30, 2019 2:32 pm
CCRL Chess Engine Testing Standards
Your post is just disrespectful and downright oblivious. The CCRL team could easily update that from "Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz)" to "Equivalent to 40 moves in 10 minutes on Ryzen 1950x (3.2 GHz)", or whatever the time scaling becomes. In fact, they even added the following "about 15 minutes on a modern Intel CPU". Which is a bit vague, but still gets the point across.

The CCRL list allows us to compare modern engines to engines written many years ago. Unless you want the CCRL team to either replay, or throw out hundreds of thousands of games that establish connections between things like Rybka and today's best, the hash size and CPU counts are there to stay.

I would think that you, as a fan of Leela + friends, would show some appreciation that the CCRL team has added GPU testing to their list.

CCRL is, and will continune to be, in my view, the single greatest list for ranking chess engines that is out there today.

Modern Times
Posts: 2410
Joined: Thu Jun 07, 2012 9:02 pm

Re: CCRL Chess Engine Match Standards. How obsolete are they?

Post by Modern Times » Sun Jun 30, 2019 4:46 pm

AndrewGrant wrote:
Sun Jun 30, 2019 4:26 pm
The CCRL team could easily update that from "Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz)" to "Equivalent to 40 moves in 10 minutes on Ryzen 1950x (3.2 GHz)", or whatever the time scaling becomes. In fact, they even added the following "about 15 minutes on a modern Intel CPU". Which is a bit vague, but still gets the point across.
Yeah we should probably think about re-naming it properly. CEGT did exactly that a few years ago. Their 40/20 list used to be "40/40 on an Athlon 64 X2 4200+".
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

mwyoung
Posts: 1641
Joined: Wed May 12, 2010 8:00 pm

Re: CCRL Chess Engine Match Standards. How obsolete are they?

Post by mwyoung » Sun Jun 30, 2019 9:53 pm

xr_a_y wrote:
Sun Jun 30, 2019 4:07 pm
So what ?, 40/40 means something like 40/10 and 40/4 means something like 40/1. This is not an issue I think, CCRL testers are using heterogeneous hardware anyway, so "scaling" is alwas needed.

Do you really want engines to be tested on 8 threads @3.8GHz, 32Gb hash for 40 min TC ? For what purpose ?
The times have changed. You can not test the A/B engines on a system setting that performs less then a smartphone. Without proper testing methods. The NN engines can not be tested correctly. This is why Lc0 rating and ranking is so wrong by CCRL. NN engines have to have modern hardware. This is not the year 2005 anymore.


1 Stockfish 10 64-bit 4CPU 3546 +13 −12 69.6% −124.9 54.9% 2015
100.0%
2 Houdini 6 64-bit 4CPU 3519 +9 −9 65.5% −108.4 53.9% 3912
95.8%
3 Komodo 11.2 64-bit 4CPU 3503 +16 −16 58.2% −66.6 55.3% 1158
90.4%
4 Lc0 0.21.1 JH.T6.532 GPU 3487 +17 −17 59.2% −58.5 52.4% 1100
100.0
Professing themselves to be wise, they became fools,
Take on me. foes 0

elcabesa
Posts: 815
Joined: Sun May 23, 2010 11:32 am
Contact:

Re: CCRL Chess Engine Match Standards. How obsolete are they?

Post by elcabesa » Sun Jun 30, 2019 10:11 pm

so the real problem for you is that LC0 and NN behave so poorly?

AndrewGrant
Posts: 490
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: CCRL Chess Engine Match Standards. How obsolete are they?

Post by AndrewGrant » Sun Jun 30, 2019 10:21 pm

elcabesa wrote:
Sun Jun 30, 2019 10:11 pm
so the real problem for you is that LC0 and NN behave so poorly?
That tends to be the real motivation for those with an obsession with LC0.

mwyoung
Posts: 1641
Joined: Wed May 12, 2010 8:00 pm

Re: CCRL Chess Engine Match Standards. How obsolete are they?

Post by mwyoung » Sun Jun 30, 2019 10:25 pm

elcabesa wrote:
Sun Jun 30, 2019 10:11 pm
so the real problem for you is that LC0 and NN behave so poorly?
Is it really correct to advertise a rating list as 4m/40moves. When CCRL is really testing at 1.5m/40 moves.
Is it really correct to advertise a rating list as 40m/40moves When CCRL is really testing at 15m/40 moves.

And it is not 2005 anymore. You can not test the modern NN engines without proper hardware.
And you can not have a fair match to test the NN engines.
If the A/B are running a system less powerful then a modern smartphone.
Professing themselves to be wise, they became fools,
Take on me. foes 0

mwyoung
Posts: 1641
Joined: Wed May 12, 2010 8:00 pm

Re: CCRL Chess Engine Match Standards. How obsolete are they?

Post by mwyoung » Sun Jun 30, 2019 10:41 pm

AndrewGrant wrote:
Sun Jun 30, 2019 10:21 pm
elcabesa wrote:
Sun Jun 30, 2019 10:11 pm
so the real problem for you is that LC0 and NN behave so poorly?
That tends to be the real motivation for those with an obsession with LC0.
I have an obsession with testing chess computers since 1982.
And some of you A/B guys really hate Lc0.

And my favorite engines is always the best tested engines.

Some may remember when Houdini was best, and I tested Stockfish as best.
The same kind of fan boys said I had an obsession with Stockfish.
I called it as Stockfish was tested. The Best....Now Lc0 is best by test.
Professing themselves to be wise, they became fools,
Take on me. foes 0

Post Reply