UCI_Elo 1500 engines and FIDE Elo

[Ras]

Some engines are added. The top 3 are now CT800, Delfi and Stockfish.

Thanks for testing! Btw., the CT800 engine automatically scales up or down within a +/- 50 Elo window depending on the move time because humans make disproportionally more mistakes at short time controls.

[emadsen]

Thanks Ferdy. Very useful info. Great idea to calibrate engine-engine ratings from engine-human ratings gathered from Lichess.

Looks like MadChess plays too well for a given Elo setting. I have some work to do on UCI_LimitStrength for the next release...

[lkaufman]

Maia Lichess 1546 has a blitz Lichess rating of around 1546. This engine is run with maia-1100.pb.gz net at node 1 on a gauntlet test below.

The formula to get FIDE Elo from Lichess Blitz Elo is,

Code: Select all

FIDE rating = 65 + [0.87 x Lichess Blitz Rating]

based from the regression in 2019.

Maia FIDE Elo = 65 + 0.87 x 1546 = 1410

Maia gauntlet bullet at TC 60s+100ms and is set at FIDE Elo 1410.

Code: Select all

   # Player                                    :  FIDE Elo  Points  Played  Points(%)
   1 Rybka v2.3.2a UCI_Elo 1500                :      2217    99.0     100       99.0
   2 Minic 2.51 UCI_Elo 1500                   :      2145    98.5     100       98.5
   3 Cheese 2.1 UCI_Elo 1500                   :      2094    98.0     100       98.0
   4 Honey X5i UCI_Elo 1500                    :      2054    97.5     100       97.5
   5 Cheng 4.39 UCI_Elo 1500                   :      2054    97.5     100       97.5
   6 Ufim v8.02 UCI_Elo 1500                   :      2021    97.0     100       97.0
   7 Amyan 1.72 UCI_Elo 1500                   :      1910    94.5     100       94.5
   8 Wasp 4.5 UCI_Elo 1500                     :      1840    92.0     100       92.0
   9 MadChess 3.0 UCI_Elo 1500                 :      1817    91.0     100       91.0
  10 Deuterium v2021.1.38.29.4 UCI_Elo 1500    :      1753    87.5     100       87.5
  11 Rhetoric 1.4.3 UCI_Elo 1500               :      1737    86.5     100       86.5
  12 Discocheck 5.2 UCI_Elo 1500               :      1730    86.0     100       86.0
  13 Arasan 21.1 UCI_Elo 1500                  :      1716    85.0     100       85.0
  14 Rodent IV 022 UCI_Elo 1500                :      1716    85.0     100       85.0
  15 Deuterium v2019.2.37.73 UCI_Elo 1500      :      1599    74.5     100       74.5
  16 Danasah 7.9 UCI_Elo 1500                  :      1599    74.5     100       74.5
  17 Stockfish 13 UCI_Elo 1500                 :      1539    67.5     100       67.5
  18 Hiarcs 14 UCI_Elo 1500                    :      1449    55.5     100       55.5
  19 Maia Lichess 1546                         :      1410   312.5    1900       16.4
  20 Deuterium v2014.2.33.276 UCI_Elo 1500     :      1171    20.5     100       20.5

The closest engines at FIDE Elo 1500 within 100 Elo are Stockfish, Hiarcs, Maia, Danasah and Deuterium v2019.2.37.73.

Looks like several engines are tuning to CCRL 1500 which is significantly stronger than 1500 FIDE . about 1890 +/- 100

Does this last sentence mean that on average, an engine with a CCRL blitz rating of 1500 should be an even match for a human with a FIDE rating of about 1890 if they are playing blitz chess? If so what is the time limit assumed for the human vs engine match? I suppose it is based on the average blitz game played between humans and engines on Lichess, is that correct? If so can you estimate the average time limit of such human vs engine games? Time limit makes a huge difference in engine vs human play; a human might be hundreds of elo stronger against engines at game in ten minutes than at game in three minutes. This estimate of about four hundred elo gap between CCRL and FIDE ratings in blitz might be about right at some time control in that range.

[KLc]

I have created the following list. The Elo was computed via Ordo and I have anchored Shredder with UCI_Elo 1300 at Elo 1300. Except for HIARCS, all engines produce a move independently of time control. For HIARCS I used 3s per move on average over a 10 move sequence (which is something in the ballpark of 40/2m).

Code: Select all

   # PLAYER                              :  RATING  POINTS  PLAYED   (%)
   1 Dragon (Skill 13)                   :  1643.1   291.0     318    92
   2 Deep Shredder 12 (Elo 1500)         :  1558.5   277.5     318    87
   3 Sargon 1978 V1.01 (Depth 3)         :  1511.9   268.5     318    84
   4 Arasan 22.1 (Elo 1300)              :  1386.8   238.0     318    75
   5 Dragon (Skill 11)                   :  1360.0   231.0     318    73
   6 Arasan 22.1 (Elo 1500)              :  1343.2   225.0     317    71
   7 Maia Chess (Elo 1500)               :  1337.9   223.0     317    70
   8 Deep Shredder 12 (Elo 1300)         :  1300.0   213.0     318    67
   9 HIARCS 14 WCSC (Elo 1300, 3s/ma)    :  1288.0   102.0     156    65
  10 Sargon 1978 V1.01 (Depth 2)         :  1272.1   204.0     318    64
  11 Arasan 22.1 (Elo 1100)              :  1220.7   188.0     318    59
  12 HIARCS 14 WCSC (Elo 1500, 3s/ma)    :  1210.2    90.0     156    58
  13 Maia Chess (Elo 1300)               :  1199.6   180.0     318    57
  14 ChessPuter Feb18                    :  1115.6   151.5     318    48
  15 Maia Chess (Elo 1100)               :  1087.7   141.0     318    44
  16 HIARCS 14 WCSC (Elo 1100, 3s/ma)    :  1083.0    69.5     156    45
  17 Dragon (Skill 9)                    :  1081.8   140.0     318    44
  18 PyTuroChamp Feb20                   :  1076.3   137.5     318    43
  19 Bernstein Feb20                     :  1044.4   127.0     318    40
  20 Sargon 1978 V1.01 (Depth 1)         :  1018.9   118.5     318    37
  21 ChessEngine by A. Lou May19         :  1014.8   117.0     318    37
  22 Deep Shredder 12 (Elo 1100)         :  1012.5   116.0     318    36
  23 Andoma Feb21                        :  1007.0   114.5     317    36
  24 Plankalkül Feb20                    :   975.0   103.5     317    33
  25 Alouette 0.1.4                      :   947.7    96.5     318    30
  26 Shannon Feb20                       :   917.7    87.5     317    28
  27 SOMA Feb20                          :   843.5    67.5     317    21
  28 Acqua 2.0                           :   725.5    43.0     318    14
  29 Random Mover Feb20                  :   259.3     3.5     318     1

[Carlos777]

Interesting. Although I did not understand in what conditions the engines played. Some used depth like Sargon and you already mentioned Hiarcs, what about the others?

OTOH, where can I download the following engines?

Bernstein
Andoma
Panklalkul
Shannon
SOMA

Regards,
Carlos

I have created the following list. The Elo was computed via Ordo and I have anchored Shredder with UCI_Elo 1300 at Elo 1300. Except for HIARCS, all engines produce a move independently of time control. For HIARCS I used 3s per move on average over a 10 move sequence (which is something in the ballpark of 40/2m).

[Ferdy]

Maia gauntlet bullet at TC 60s+100ms and is set at FIDE Elo 1410.

Code: Select all

   # Player                                    :  FIDE Elo  Points  Played  Points(%)
   1 Rybka v2.3.2a UCI_Elo 1500                :      2217    99.0     100       99.0
   2 Minic 2.51 UCI_Elo 1500                   :      2145    98.5     100       98.5
   3 Cheese 2.1 UCI_Elo 1500                   :      2094    98.0     100       98.0
   4 Honey X5i UCI_Elo 1500                    :      2054    97.5     100       97.5
   5 Cheng 4.39 UCI_Elo 1500                   :      2054    97.5     100       97.5
   6 Ufim v8.02 UCI_Elo 1500                   :      2021    97.0     100       97.0
   7 Amyan 1.72 UCI_Elo 1500                   :      1910    94.5     100       94.5
   8 Wasp 4.5 UCI_Elo 1500                     :      1840    92.0     100       92.0
   9 MadChess 3.0 UCI_Elo 1500                 :      1817    91.0     100       91.0
  10 Deuterium v2021.1.38.29.4 UCI_Elo 1500    :      1753    87.5     100       87.5
  11 Rhetoric 1.4.3 UCI_Elo 1500               :      1737    86.5     100       86.5
  12 Discocheck 5.2 UCI_Elo 1500               :      1730    86.0     100       86.0
  13 Arasan 21.1 UCI_Elo 1500                  :      1716    85.0     100       85.0
  14 Rodent IV 022 UCI_Elo 1500                :      1716    85.0     100       85.0
  15 Deuterium v2019.2.37.73 UCI_Elo 1500      :      1599    74.5     100       74.5
  16 Danasah 7.9 UCI_Elo 1500                  :      1599    74.5     100       74.5
  17 Stockfish 13 UCI_Elo 1500                 :      1539    67.5     100       67.5
  18 Hiarcs 14 UCI_Elo 1500                    :      1449    55.5     100       55.5
  19 Maia Lichess 1546                         :      1410   312.5    1900       16.4
  20 Deuterium v2014.2.33.276 UCI_Elo 1500     :      1171    20.5     100       20.5

The closest engines at FIDE Elo 1500 within 100 Elo are Stockfish, Hiarcs, Maia, Danasah and Deuterium v2019.2.37.73.

The hardware I used, i7-2600K 3.4Ghz 4-core/8-threads, to run the matches has a stockfish_13_win_x64 bench of:

Code: Select all

Total time (ms) : 4603
Nodes searched  : 3766422
Nodes/second    : 818253

The matches are conducted with cutechess-cli at 6 concurrent games and all engines are un in a single thread.

[Ferdy]

Maia Lichess 1546 has a blitz Lichess rating of around 1546. This engine is run with maia-1100.pb.gz net at node 1 on a gauntlet test below.

The Maia Lichess 1546 blitz rating has now changed. It is not 1546 anymore so the rating diff of 1500-Elo may now change.

Blitz Rating history:



Better still just run your engine in lichess at UCI_Elo 1500, simulating the hardware of CCRL or CEGT as much as possible to also get an estimate of rating conversion.

[KLc]

Interesting. Although I did not understand in what conditions the engines played. Some used depth like Sargon and you already mentioned Hiarcs, what about the others?

They move by some fixed conditions, mostly depth or number of nodes. You can basically say "go depth 99" and they will just do their thing and spit out a move.

OTOH, where can I download the following engines?

Bernstein
Andoma
Panklalkul
Shannon
SOMA

All except Andoma are part of the excellent PyTuroChamp project: https://github.com/mdoege/PyTuroChamp. Andoma is from https://github.com/healeycodes/andoma. I like to crawl for simple engines and try my best to beat them.

[Guenther]

Interesting. Although I did not understand in what conditions the engines played. Some used depth like Sargon and you already mentioned Hiarcs, what about the others?

OTOH, where can I download the following engines?

Bernstein
Andoma
Panklalkul
Shannon
SOMA

Regards,
Carlos

Those are PyturoChamp personalities (except Andoma which I tested two months ago).

[wheelman]

Some engines are added. The top 3 are now CT800, Delfi and Stockfish.

Code: Select all

 num                                  engine games  fide_elo  diff
   1                CT800 V1.42 UCI_Elo 1500   100      1515    15
   2                  Delfi 5.4 UCI_Elo 1500   100      1463    37
   3               Stockfish 13 UCI_Elo 1500   100      1539    39
   4                  Hiarcs 14 UCI_Elo 1500   100      1449    51
   5                       Maia Lichess 1546  2500      1410    90
   6                Danasah 7.9 UCI_Elo 1500   100      1599    99
   7    Deuterium v2019.2.37.73 UCI_Elo 1500   100      1599    99
   8              Rodent IV 022 UCI_Elo 1500   100      1716   216
   9                Arasan 21.1 UCI_Elo 1500   100      1716   216
  10                 Cheng 4.40 UCI_Elo 1500   100      1730   230
  11             Discocheck 5.2 UCI_Elo 1500   100      1730   230
  12             Rhetoric 1.4.3 UCI_Elo 1500   100      1737   237
  13  Deuterium v2021.1.38.29.4 UCI_Elo 1500   100      1753   253
  14               MadChess 3.0 UCI_Elo 1500   100      1817   317
  15   Deuterium v2014.2.33.276 UCI_Elo 1500   100      1171   329
  16                   Wasp 4.5 UCI_Elo 1500   100      1840   340
  17                 Amyan 1.72 UCI_Elo 1500   100      1910   410
  18                 Ufim v8.02 UCI_Elo 1500   100      2021   521
  19                 Cheng 4.39 UCI_Elo 1500   100      2054   554
  20                  Honey X5i UCI_Elo 1500   100      2054   554
  21                 Cheese 2.1 UCI_Elo 1500   100      2094   594
  22                 Cheese 2.2 UCI_Elo 1500   100      2146   646
  23                 Minic 2.51 UCI_Elo 1500   100      2146   646
  24              Rybka v2.3.2a UCI_Elo 1500   100      2217   717
  25                  Spike 1.4 UCI_Elo 1500   100      2400   900
  26               Equinox 3.30 UCI_Elo 1500   100      2400   900


diff : The absolute difference between 1500 and fide_elo, lower diff is better.

I have an off-topic question if you don't mind. How do you get Delfi 5.4 to play at 1500 ELO? Do you have a version different from the free version? Thank you.