CCRL Chess Engine Testing Standards..jpg
CCRL Chess Engine Match Standards. How obsolete are they?
Moderators: hgm, Rebel, chrisw
-
- Posts: 2727
- Joined: Wed May 12, 2010 10:00 pm
CCRL Chess Engine Match Standards. How obsolete are they?
"The worst thing that can happen to a forum is a running wild attacking moderator(HGM) who is not corrected by the community." - Ed Schröder
But my words like silent raindrops fell. And echoed in the wells of silence.
But my words like silent raindrops fell. And echoed in the wells of silence.
-
- Posts: 855
- Joined: Sun May 23, 2010 1:32 pm
Re: CCRL Chess Engine Match Standards. How obsolete are they?
I think it mainly means "consistency"
-
- Posts: 1871
- Joined: Sat Nov 25, 2017 2:28 pm
- Location: France
Re: CCRL Chess Engine Match Standards. How obsolete are they?
So what ?, 40/40 means something like 40/10 and 40/4 means something like 40/1. This is not an issue I think, CCRL testers are using heterogeneous hardware anyway, so "scaling" is alwas needed.
Do you really want engines to be tested on 8 threads @3.8GHz, 32Gb hash for 40 min TC ? For what purpose ?
Do you really want engines to be tested on 8 threads @3.8GHz, 32Gb hash for 40 min TC ? For what purpose ?
-
- Posts: 1766
- Joined: Tue Apr 19, 2016 6:08 am
- Location: U.S.A
- Full name: Andrew Grant
Re: CCRL Chess Engine Match Standards. How obsolete are they?
Your post is just disrespectful and downright oblivious. The CCRL team could easily update that from "Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz)" to "Equivalent to 40 moves in 10 minutes on Ryzen 1950x (3.2 GHz)", or whatever the time scaling becomes. In fact, they even added the following "about 15 minutes on a modern Intel CPU". Which is a bit vague, but still gets the point across.
The CCRL list allows us to compare modern engines to engines written many years ago. Unless you want the CCRL team to either replay, or throw out hundreds of thousands of games that establish connections between things like Rybka and today's best, the hash size and CPU counts are there to stay.
I would think that you, as a fan of Leela + friends, would show some appreciation that the CCRL team has added GPU testing to their list.
CCRL is, and will continune to be, in my view, the single greatest list for ranking chess engines that is out there today.
#WeAreAllDraude #JusticeForDraude #RememberDraude #LeptirBigUltra
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
-
- Posts: 3554
- Joined: Thu Jun 07, 2012 11:02 pm
Re: CCRL Chess Engine Match Standards. How obsolete are they?
Yeah we should probably think about re-naming it properly. CEGT did exactly that a few years ago. Their 40/20 list used to be "40/40 on an Athlon 64 X2 4200+".AndrewGrant wrote: ↑Sun Jun 30, 2019 6:26 pm The CCRL team could easily update that from "Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz)" to "Equivalent to 40 moves in 10 minutes on Ryzen 1950x (3.2 GHz)", or whatever the time scaling becomes. In fact, they even added the following "about 15 minutes on a modern Intel CPU". Which is a bit vague, but still gets the point across.
-
- Posts: 2727
- Joined: Wed May 12, 2010 10:00 pm
Re: CCRL Chess Engine Match Standards. How obsolete are they?
The times have changed. You can not test the A/B engines on a system setting that performs less then a smartphone. Without proper testing methods. The NN engines can not be tested correctly. This is why Lc0 rating and ranking is so wrong by CCRL. NN engines have to have modern hardware. This is not the year 2005 anymore.xr_a_y wrote: ↑Sun Jun 30, 2019 6:07 pm So what ?, 40/40 means something like 40/10 and 40/4 means something like 40/1. This is not an issue I think, CCRL testers are using heterogeneous hardware anyway, so "scaling" is alwas needed.
Do you really want engines to be tested on 8 threads @3.8GHz, 32Gb hash for 40 min TC ? For what purpose ?
1 Stockfish 10 64-bit 4CPU 3546 +13 −12 69.6% −124.9 54.9% 2015
100.0%
2 Houdini 6 64-bit 4CPU 3519 +9 −9 65.5% −108.4 53.9% 3912
95.8%
3 Komodo 11.2 64-bit 4CPU 3503 +16 −16 58.2% −66.6 55.3% 1158
90.4%
4 Lc0 0.21.1 JH.T6.532 GPU 3487 +17 −17 59.2% −58.5 52.4% 1100
100.0
"The worst thing that can happen to a forum is a running wild attacking moderator(HGM) who is not corrected by the community." - Ed Schröder
But my words like silent raindrops fell. And echoed in the wells of silence.
But my words like silent raindrops fell. And echoed in the wells of silence.
-
- Posts: 855
- Joined: Sun May 23, 2010 1:32 pm
Re: CCRL Chess Engine Match Standards. How obsolete are they?
so the real problem for you is that LC0 and NN behave so poorly?
-
- Posts: 1766
- Joined: Tue Apr 19, 2016 6:08 am
- Location: U.S.A
- Full name: Andrew Grant
Re: CCRL Chess Engine Match Standards. How obsolete are they?
That tends to be the real motivation for those with an obsession with LC0.
#WeAreAllDraude #JusticeForDraude #RememberDraude #LeptirBigUltra
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
-
- Posts: 2727
- Joined: Wed May 12, 2010 10:00 pm
Re: CCRL Chess Engine Match Standards. How obsolete are they?
Is it really correct to advertise a rating list as 4m/40moves. When CCRL is really testing at 1.5m/40 moves.
Is it really correct to advertise a rating list as 40m/40moves When CCRL is really testing at 15m/40 moves.
And it is not 2005 anymore. You can not test the modern NN engines without proper hardware.
And you can not have a fair match to test the NN engines.
If the A/B are running a system less powerful then a modern smartphone.
"The worst thing that can happen to a forum is a running wild attacking moderator(HGM) who is not corrected by the community." - Ed Schröder
But my words like silent raindrops fell. And echoed in the wells of silence.
But my words like silent raindrops fell. And echoed in the wells of silence.
-
- Posts: 2727
- Joined: Wed May 12, 2010 10:00 pm
Re: CCRL Chess Engine Match Standards. How obsolete are they?
I have an obsession with testing chess computers since 1982.AndrewGrant wrote: ↑Mon Jul 01, 2019 12:21 amThat tends to be the real motivation for those with an obsession with LC0.
And some of you A/B guys really hate Lc0.
And my favorite engines is always the best tested engines.
Some may remember when Houdini was best, and I tested Stockfish as best.
The same kind of fan boys said I had an obsession with Stockfish.
I called it as Stockfish was tested. The Best....Now Lc0 is best by test.
"The worst thing that can happen to a forum is a running wild attacking moderator(HGM) who is not corrected by the community." - Ed Schröder
But my words like silent raindrops fell. And echoed in the wells of silence.
But my words like silent raindrops fell. And echoed in the wells of silence.