CCRL 40/40, 40/4 and FRC lists updated (23rd February 2019)

[Graham Banks]

The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.
Also, please note that the 40/40 and 40/4 lists are constructed from totally different databases and not comparable to each other.

Thanks to all of our currently active testers (Graham, Ray, Tirsa, Gabor, Sergio, Andreas and Brent), plus to Chessdom.com for providing an online server to assist with resources.

40/40 testing this week that I'm aware of will include (with live broadcast port for TLCV noted where applicable):

73rd Amateur Series Division 1 Tournament (finishing Friday 16002)
73rd Amateur Series Division 2 Tournament (continuing 16001)
73rd Amateur Series Division 3 Tournament (starting Friday 16002)
Jumbo And The Monster Tournament (continuing 16053)
Asymptote 0.4.1 64-bit Gauntlet (continuing 16063)
Topple 0.3.5 64-bit Gauntlet (continuing 16064)
Jumbo 0.6.96 64-bit Gauntlet (continuing 16065)
SugaR 1.6 64-bit Gauntlet (finishing Thursday 16091)
Laser 1.7 64-bit Gauntlet (continuing 16092)
Ethereal 11.25 64-bit Gauntlet (continuing 16093)
RubiChess 1.3 64-bit Gauntlet (continuing 16094)
The Red Jewel 4CPU Tournament (continuing 16083)
Tongues Of Fire 4CPU Tournament (continuing 16084)
RofChade 2.0 64-bit 4CPU Gauntlet (being run by Tirsa)
Minic 0.40 64-bit Gauntlet (being run by Tirsa)
Barbarossa 0.5.0 64-bit Gauntlet (being run by Tirsa)
Various other 1CPU Gauntlets (being run by Tirsa)
Francesca MAD 0.21 64-bit Gauntlet (being run by Gabor)
Deuterium 2019.1.36.50 64-bit Gauntlet (being run by Gabor)
Various 1CPU Tournaments (being run by Sergio and Andreas)

40/4 testing since the last update report has included (thanks to Gabor and Sergio):

Laser 1.7 64-bit
Robocide 0.4 64-bit
Ethereal 11.25 64-bit 4CPU
Laser 1.7 64-bit 4CPU
Asymptote 0.4.2 64-bit
Bumblebee 1.0 64-bit
Minic 0.34 64-bit
RubiChess 1.3 64-bit
Franky 1.0 64-bit
Topple 0.3.4 64-bit
ECE-X3.5 64-bit
Various Tournaments (run by Sergio)

FRC 40/4 testing since the last update report has included (thanks to Ray):

Arminius 2018-12-23 64-bit

[Graham Banks]

Brent did some testing using the time ratio and if the results had been included, Lc0 GPU would slot into 3rd spot, below SF and Houdini on the CCRL 40/4 list.

[Modern Times]

I'm keen that we get GPU testing with Lc0 and others on our lists and I'm certain we will at some point. But is has to be done the right way, and we could not agree with Brent on the approach to take. So we wait a bit longer !

None of our current testers have decent GPUs, because in the past when building a chess machine you could choose a basic GPU and put the money into the CPU and the rest of the machine instead. Perhaps once we get into hardware replacement cycles that will be the time, and with luck GPU prices will come down as well.

[Modern Times]

On CEGT 40/4, comparing to 4CPU engines, Lc0 on GPU is in 4th place behind Stockfish, Houdini and Komodo. I'm not sure what GPU they use.

[konsolas]

Possible suggestion:

The GTX 1050 has a low TDP doesn't use any additional PCI-E power connectors so it would probably be an easy upgrade for most of your existing systems (as long as they're not laptops).

In the UK, it's priced very similarly to a modern quad core (the i3 8100 costs £113, the 1050 costs £117), so it may be a fair comparison with CCRL 4CPU test conditions.

[Modern Times]

Possible suggestion:

The GTX 1050 has a low TDP doesn't use any additional PCI-E power connectors so it would probably be an easy upgrade for most of your existing systems (as long as they're not laptops).

In the UK, it's priced very similarly to a modern quad core (the i3 8100 costs £113, the 1050 costs £117), so it may be a fair comparison with CCRL 4CPU test conditions.

Thanks for the suggestion, much appreciated - so when you say "no additional connectors" does that mean none at all, it draws it all from the PCI-E slot ? Or it needs one connector ? It has been so long since I bought CPU components and built a machine I am completely clueless now. At least on this machine I'm pretty sure I have one connector from memory, 1x 6-pin (ATI Radeon HD 6850 from 2011 !!)

Edit: Ah I see: "Power Supply: Powered directly from the PCIe x16 slot, no additional PCIe connector necessary"

For the 1050 would you need a 2048Mb card or a 4096MB ?

[konsolas]

Possible suggestion:

The GTX 1050 has a low TDP doesn't use any additional PCI-E power connectors so it would probably be an easy upgrade for most of your existing systems (as long as they're not laptops).

In the UK, it's priced very similarly to a modern quad core (the i3 8100 costs £113, the 1050 costs £117), so it may be a fair comparison with CCRL 4CPU test conditions.

Thanks for the suggestion, much appreciated - so when you say "no additional connectors" does that mean none at all, it draws it all from the PCI-E slot ? Or it needs one connector ? It has been so long since I bought CPU components and built a machine I am completely clueless now. At least on this machine I'm pretty sure I have one connector from memory, 1x 6-pin (ATI Radeon HD 6850 from 2011 !!)

For the 1050 would you need a 2048Mb card or a 4096MB ?

Yes, it uses no additional connectors at all and draws all of its power from the PCI-E slot. It only uses 75W. This shouldn't be a problem with most motherboards, but some PCI-E slots may have a reduced power limit.

The GTX 1050 has 2GB of GDDR5, although there is also a 3GB version, but that one does not appear to be widely available.

The 4GB model you're talking about is actually a 1050Ti, which is a bit faster and a bit more expensive at around £130 to £160 in my country.
The 1050Ti is the fastest GPU that NVIDIA sells (excluding the Quadros) which does not require additional power connectors. It also has a TDP of 75W.

[Modern Times]

Seriously tempted by this (1050) at those prices, whilst waiting for the higher performance cards to be more affordable. Do they work in PCIe 2.0 x16 slots, and does that affect performance ?

Edit: Sorry, googled this and yes it would run, and it seems with little or no performance effect with a card at this level

[Modern Times]

Possible suggestion:

The GTX 1050 has a low TDP doesn't use any additional PCI-E power connectors so it would probably be an easy upgrade for most of your existing systems (as long as they're not laptops).

In the UK, it's priced very similarly to a modern quad core (the i3 8100 costs £113, the 1050 costs £117), so it may be a fair comparison with CCRL 4CPU test conditions.

I ordered an ASUS GTX 1050 card today, £100 delivered. Given that our reference CPU is an ancient Athlon64 4600+, the GTX 1050 should be more than adequate and very fair. The good folks at CEGT, who are already doing GPU testing, have been very helpful and giving me advice on set-up and how they do their testing. Hopefully this weekend I'll be up and running with it.

[jorose]

Possible suggestion:

The GTX 1050 has a low TDP doesn't use any additional PCI-E power connectors so it would probably be an easy upgrade for most of your existing systems (as long as they're not laptops).

In the UK, it's priced very similarly to a modern quad core (the i3 8100 costs £113, the 1050 costs £117), so it may be a fair comparison with CCRL 4CPU test conditions.

I ordered an ASUS GTX 1050 card today, £100 delivered. Given that our reference CPU is an ancient Athlon64 4600+, the GTX 1050 should be more than adequate and very fair. The good folks at CEGT, who are already doing GPU testing, have been very helpful and giving me advice on set-up and how they do their testing. Hopefully this weekend I'll be up and running with it.

In general I think it's great that you guys want to test engines with GPU, but there are some issues that I think should at least be discussed.
Are you intending to only ever have a single GPU model you test with, or how would you rescale to ensure consistency between testers? If both CPU and GPU differ between testers, how would you rescale between them?

Also, by what metric would the GTX 1050 be in any way a fair comparison to the 13 year old reference CPU? I understand that if we used GPUs from 2006 then basically there wouldn't be a benefit at all (no real NN support), but maybe it is time to upgrade the reference CPU in that case?

Im aware that these may not be solvable problems, but to me that is a sign that perhaps a new list is needed, not that the issues should get ignored.