Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

smatovic
Posts: 2634
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic
Contact:

Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by smatovic »

Heyho, I already mentioned it in the programmers section, for comparing engines
over computer chess history or different architectures I propose a metric like:

Elo / (Transistorcount*Frequency)

Can be applied backwards with electro-mechanical relays, vacuum-tubes,
transistors, ICs and microchips.

Can be applied sidewards with CPU, VPU, ASIC, FPGA, GPU, TPU.

Can be applied with or w/o memory (e.g. SRAM or DRAM, no core rope memory?).

Dunno how to be applied forward for quantum-gates, memristors and alike.

--
Srdja
User avatar
towforce
Posts: 11525
Joined: Thu Mar 09, 2006 12:57 am
Location: Birmingham UK

Re: Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by towforce »

Hmmmmm....

If I call transistors*frequency "computer power" (and accepting that extra transistors are not actually directly equivalent to a higher clock frequency - though I think there's probably a good correlation between GeekBench score and elo score - so for me "GeekBench score" would be better than transistors*frequency), then I would have thought that the thing you really want to know is...

Elo for specified program at given computer power.

For a given program, elo for given computer power is likely to be logarithmic if you're lucky, and have a hard upper limit if you're not lucky.

So with computer power on the x axis and elo on the y axis, it's likely to have this shape (ignore the actual numbers on the y axis, or multiply them by 1000):

Image

Big breakthroughs that meaningfully move the curve upwards are becoming increasingly rare: the last one was the application of ANNs to the task, following Google's breakthrough in Go with AlphaGo, and people working out how to apply that to chess.

These major breakthroughs (sometimes called "paradigm shifts") have always been more interesting to me than techniques which yield marginal gains. Unfortunately for me, most members of this forum prefer to discuss the marginal gains.
Writing is the antidote to confusion.
It's not "how smart you are", it's "how are you smart".
Your brain doesn't work the way you want, so train it!
smatovic
Posts: 2634
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic
Contact:

Re: Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by smatovic »

towforce wrote: Sat Nov 26, 2022 3:13 pm Hmmmmm....

If I call transistors*frequency "computer power" (and accepting that extra transistors are not actually directly equivalent to a higher clock frequency - though I think there's probably a good correlation between GeekBench score and elo score - so for me "GeekBench score" would be better than transistors*frequency), then I would have thought that the thing you really want to know is...
Hmm, does GeekBench run on the 6502, IBM DeepBlue, Hydra, Google TPU, etc.?
towforce wrote: Sat Nov 26, 2022 3:13 pm Elo for specified program at given computer power.

For a given program, elo for given computer power is likely to be logarithmic if you're lucky, and have a hard upper limit if you're not lucky.

So with computer power on the x axis and elo on the y axis, it's likely to have this shape (ignore the actual numbers on the y axis, or multiply them by 1000):

Image

Big breakthroughs that meaningfully move the curve upwards are becoming increasingly rare: the last one was the application of ANNs to the task, following Google's breakthrough in Go with AlphaGo, and people working out how to apply that to chess.

These major breakthroughs (sometimes called "paradigm shifts") have always been more interesting to me than techniques which yield marginal gains. Unfortunately for me, most members of this forum prefer to discuss the marginal gains.
The funny thing is, Lc0 is derived from A0 is derived from AlphaGo, MCTS-PUCT with CNNs came from the Go world to chess, SF AB NNUE came to chess from the Shogi world...I am not that much into NNs with NNUE, but my take for the next Elo jump would be NNOM++ - Move Ordering Neural Networks:

https://talkchess.com/forum3/viewtopic.php?f=7&t=80364

--
Srdja
smatovic
Posts: 2634
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic
Contact:

Re: Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by smatovic »

towforce wrote: Sat Nov 26, 2022 3:13 pm [...]
Elo for specified program at given computer power.

For a given program, elo for given computer power is likely to be logarithmic if you're lucky, and have a hard upper limit if you're not lucky.

So with computer power on the x axis and elo on the y axis, it's likely to have this shape (ignore the actual numbers on the y axis, or multiply them by 1000):
[...]
Hmm, keep in mind, there is also a space-time tradeoff and knowledge-search tradeoff going on, imagine 32-men EGTB, imagine TB neural networks....

--
Srdja
User avatar
towforce
Posts: 11525
Joined: Thu Mar 09, 2006 12:57 am
Location: Birmingham UK

Re: Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by towforce »

smatovic wrote: Sat Nov 26, 2022 10:03 amDunno how to be applied forward for quantum-gates...

One qubit = one transistor, and the "frequency" is the number of times it can be put into a superposition, entangled with other qubits, collapsed from a superposition to a single state, and be measured in a second (allowing for decoherence, error correction etc).

I don't actually think you'll ever have to worry about that, though. There once was a computer with the following specification:

* 400 flops

* weighed over 27,000 kg

* big enough to park a school bus inside

* consumed 160 kw of power

* unreliable: it only worked half the time (literally!)

* programming it entailed rewiring it, which took days!

* price: $6 million in 2022 dollars

What a crap computer by today's standards! Absolute rubbish!

This was ENIAC in 1946. At the time, it was actually remarkably good - far better than anything that had gone before - and was really good for big calculations. Consequently, more computers were built, and they rapidly improved.

However, todays quantum computers are nowhere near as good as today's standard computers: not even cheap computers! So while they're a good idea, I don't think they're going to get the investment to develop like 1940s/1950s computers did.
Writing is the antidote to confusion.
It's not "how smart you are", it's "how are you smart".
Your brain doesn't work the way you want, so train it!
smatovic
Posts: 2634
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic
Contact:

Re: Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by smatovic »

towforce wrote: Sat Nov 26, 2022 5:40 pm [...]
However, todays quantum computers are nowhere near as good as today's standard computers: not even cheap computers! So while they're a good idea, I don't think they're going to get the investment to develop like 1940s/1950s computers did.
Well, at some point Moore's Law is dead, we already have GPUs with 450W and CPUs with 300+W. 3nm, 2nm, 2+nm (I know, marketing, but reflects transistorcount/mm2) in pipe, but in the long-term we need alternatives, quantum, memristors, photonics, dunno.

--
Srdja

PS:
User avatar
towforce
Posts: 11525
Joined: Thu Mar 09, 2006 12:57 am
Location: Birmingham UK

Re: Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by towforce »

smatovic wrote: Sat Nov 26, 2022 6:18 pmWell, at some point Moore's Law is dead, we already have GPUs with 450W and CPUs with 300+W. 3nm, 2nm, 2+nm (I know, marketing, but reflects transistorcount/mm2) in pipe, but in the long-term we need alternatives, quantum, memristors, photonics, dunno.

Not forgetting the topic of this thread - better software!

I know I'm in a minority here, but I am sincere in my beliefs that:

1. It's possible to play outstanding chess with a small and fast algorithm

2. While I obviously cannot prove this, I genuinely believe that the available evidence makes this more likely than not

PS:
:lol:
Writing is the antidote to confusion.
It's not "how smart you are", it's "how are you smart".
Your brain doesn't work the way you want, so train it!
User avatar
flok
Posts: 481
Joined: Tue Jul 03, 2018 10:19 am
Full name: Folkert van Heusden
Contact:

Re: Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by flok »

towforce wrote: Sat Nov 26, 2022 7:05 pm 1. It's possible to play outstanding chess with a small and fast algorithm
like early versions of rofchade
smatovic
Posts: 2634
Joined: Wed Mar 10, 2010 10:18 pm
Location: Hamburg, Germany
Full name: Srdja Matovic
Contact:

Re: Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by smatovic »

towforce wrote: Sat Nov 26, 2022 7:05 pm Not forgetting the topic of this thread - better software!

I know I'm in a minority here, but I am sincere in my beliefs that:

1. It's possible to play outstanding chess with a small and fast algorithm

2. While I obviously cannot prove this, I genuinely believe that the available evidence makes this more likely than not
I know ;)

"God's Algorithm for Chess"
https://talkchess.com/forum3/viewtopic. ... 92#p932844

--
Srdja
User avatar
towforce
Posts: 11525
Joined: Thu Mar 09, 2006 12:57 am
Location: Birmingham UK

Re: Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

Post by towforce »

flok wrote: Sat Nov 26, 2022 8:21 pm
towforce wrote: Sat Nov 26, 2022 7:05 pm 1. It's possible to play outstanding chess with a small and fast algorithm
like early versions of rofchade

It sounds as though the early versions of Rofchade achieved a very good evaluation at very low cost (in "computer power"). However, when it switched to NNUE, two things happened:

1. "computer power" required for evaluation increased

2. The overall strength increased

So: it was a very good hand-crafted evaluation (HCE), but like all the other HCEs it missed some important things. We also know that super-fast deep searching also misses important things. It is very likely that all today's NNs also miss important things.

Based on what we know about NNs, it's extremely likely that an algorithm could be both smaller (hence faster) and more accurate. Given that "death by draw" has come to the top level of correspondence chess, a top quality small/fast chess algorithm would quite likely be able to get the best available result from most positions even on a cheap computer.

Anyway - back to the topic: if early versions of Rofchade get good results at low cost, then they would get a high logarithmic curve on my chart (earlier in this thread) of elo against computer power.
Last edited by towforce on Sun Nov 27, 2022 2:59 pm, edited 2 times in total.
Writing is the antidote to confusion.
It's not "how smart you are", it's "how are you smart".
Your brain doesn't work the way you want, so train it!
Post Reply