CCRL Chess Engine Match Standards. How obsolete are they?

[mwyoung]

CCRL Chess Engine Testing Standards..jpg

[elcabesa]

I think it mainly means "consistency"

[xr_a_y]

So what ?, 40/40 means something like 40/10 and 40/4 means something like 40/1. This is not an issue I think, CCRL testers are using heterogeneous hardware anyway, so "scaling" is alwas needed.

Do you really want engines to be tested on 8 threads @3.8GHz, 32Gb hash for 40 min TC ? For what purpose ?

[AndrewGrant]

CCRL Chess Engine Testing Standards

Your post is just disrespectful and downright oblivious. The CCRL team could easily update that from "Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz)" to "Equivalent to 40 moves in 10 minutes on Ryzen 1950x (3.2 GHz)", or whatever the time scaling becomes. In fact, they even added the following "about 15 minutes on a modern Intel CPU". Which is a bit vague, but still gets the point across.

The CCRL list allows us to compare modern engines to engines written many years ago. Unless you want the CCRL team to either replay, or throw out hundreds of thousands of games that establish connections between things like Rybka and today's best, the hash size and CPU counts are there to stay.

I would think that you, as a fan of Leela + friends, would show some appreciation that the CCRL team has added GPU testing to their list.

CCRL is, and will continune to be, in my view, the single greatest list for ranking chess engines that is out there today.

[Modern Times]

The CCRL team could easily update that from "Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz)" to "Equivalent to 40 moves in 10 minutes on Ryzen 1950x (3.2 GHz)", or whatever the time scaling becomes. In fact, they even added the following "about 15 minutes on a modern Intel CPU". Which is a bit vague, but still gets the point across.

Yeah we should probably think about re-naming it properly. CEGT did exactly that a few years ago. Their 40/20 list used to be "40/40 on an Athlon 64 X2 4200+".

[mwyoung]

So what ?, 40/40 means something like 40/10 and 40/4 means something like 40/1. This is not an issue I think, CCRL testers are using heterogeneous hardware anyway, so "scaling" is alwas needed.

Do you really want engines to be tested on 8 threads @3.8GHz, 32Gb hash for 40 min TC ? For what purpose ?

The times have changed. You can not test the A/B engines on a system setting that performs less then a smartphone. Without proper testing methods. The NN engines can not be tested correctly. This is why Lc0 rating and ranking is so wrong by CCRL. NN engines have to have modern hardware. This is not the year 2005 anymore.


1 Stockfish 10 64-bit 4CPU 3546 +13 −12 69.6% −124.9 54.9% 2015
100.0%
2 Houdini 6 64-bit 4CPU 3519 +9 −9 65.5% −108.4 53.9% 3912
95.8%
3 Komodo 11.2 64-bit 4CPU 3503 +16 −16 58.2% −66.6 55.3% 1158
90.4%
4 Lc0 0.21.1 JH.T6.532 GPU 3487 +17 −17 59.2% −58.5 52.4% 1100
100.0

[elcabesa]

so the real problem for you is that LC0 and NN behave so poorly?

[AndrewGrant]

so the real problem for you is that LC0 and NN behave so poorly?

That tends to be the real motivation for those with an obsession with LC0.

[mwyoung]

so the real problem for you is that LC0 and NN behave so poorly?

Is it really correct to advertise a rating list as 4m/40moves. When CCRL is really testing at 1.5m/40 moves.
Is it really correct to advertise a rating list as 40m/40moves When CCRL is really testing at 15m/40 moves.

And it is not 2005 anymore. You can not test the modern NN engines without proper hardware.
And you can not have a fair match to test the NN engines.
If the A/B are running a system less powerful then a modern smartphone.

[mwyoung]

so the real problem for you is that LC0 and NN behave so poorly?

That tends to be the real motivation for those with an obsession with LC0.

I have an obsession with testing chess computers since 1982.
And some of you A/B guys really hate Lc0.

And my favorite engines is always the best tested engines.

Some may remember when Houdini was best, and I tested Stockfish as best.
The same kind of fan boys said I had an obsession with Stockfish.
I called it as Stockfish was tested. The Best....Now Lc0 is best by test.