Comparing Chess Engines over History or Architectures - Elo / (Transistorcount*Frequency)

[smatovic]

Heyho, I already mentioned it in the programmers section, for comparing engines
over computer chess history or different architectures I propose a metric like:

Elo / (Transistorcount*Frequency)

Can be applied backwards with electro-mechanical relays, vacuum-tubes,
transistors, ICs and microchips.

Can be applied sidewards with CPU, VPU, ASIC, FPGA, GPU, TPU.

Can be applied with or w/o memory (e.g. SRAM or DRAM, no core rope memory?).

Dunno how to be applied forward for quantum-gates, memristors and alike.

--
Srdja

[towforce]

Hmmmmm....

If I call transistors*frequency "computer power" (and accepting that extra transistors are not actually directly equivalent to a higher clock frequency - though I think there's probably a good correlation between GeekBench score and elo score - so for me "GeekBench score" would be better than transistors*frequency), then I would have thought that the thing you really want to know is...

Elo for specified program at given computer power.

For a given program, elo for given computer power is likely to be logarithmic if you're lucky, and have a hard upper limit if you're not lucky.

So with computer power on the x axis and elo on the y axis, it's likely to have this shape (ignore the actual numbers on the y axis, or multiply them by 1000):



Big breakthroughs that meaningfully move the curve upwards are becoming increasingly rare: the last one was the application of ANNs to the task, following Google's breakthrough in Go with AlphaGo, and people working out how to apply that to chess.

These major breakthroughs (sometimes called "paradigm shifts") have always been more interesting to me than techniques which yield marginal gains. Unfortunately for me, most members of this forum prefer to discuss the marginal gains.

[smatovic]

Hmmmmm....

If I call transistors*frequency "computer power" (and accepting that extra transistors are not actually directly equivalent to a higher clock frequency - though I think there's probably a good correlation between GeekBench score and elo score - so for me "GeekBench score" would be better than transistors*frequency), then I would have thought that the thing you really want to know is...

Hmm, does GeekBench run on the 6502, IBM DeepBlue, Hydra, Google TPU, etc.?

Elo for specified program at given computer power.

For a given program, elo for given computer power is likely to be logarithmic if you're lucky, and have a hard upper limit if you're not lucky.

So with computer power on the x axis and elo on the y axis, it's likely to have this shape (ignore the actual numbers on the y axis, or multiply them by 1000):



Big breakthroughs that meaningfully move the curve upwards are becoming increasingly rare: the last one was the application of ANNs to the task, following Google's breakthrough in Go with AlphaGo, and people working out how to apply that to chess.

These major breakthroughs (sometimes called "paradigm shifts") have always been more interesting to me than techniques which yield marginal gains. Unfortunately for me, most members of this forum prefer to discuss the marginal gains.

The funny thing is, Lc0 is derived from A0 is derived from AlphaGo, MCTS-PUCT with CNNs came from the Go world to chess, SF AB NNUE came to chess from the Shogi world...I am not that much into NNs with NNUE, but my take for the next Elo jump would be NNOM++ - Move Ordering Neural Networks:

https://talkchess.com/forum3/viewtopic.php?f=7&t=80364

--
Srdja

[smatovic]

[...]
Elo for specified program at given computer power.

For a given program, elo for given computer power is likely to be logarithmic if you're lucky, and have a hard upper limit if you're not lucky.

So with computer power on the x axis and elo on the y axis, it's likely to have this shape (ignore the actual numbers on the y axis, or multiply them by 1000):
[...]

Hmm, keep in mind, there is also a space-time tradeoff and knowledge-search tradeoff going on, imagine 32-men EGTB, imagine TB neural networks....

--
Srdja

[towforce]

Dunno how to be applied forward for quantum-gates...

One qubit = one transistor, and the "frequency" is the number of times it can be put into a superposition, entangled with other qubits, collapsed from a superposition to a single state, and be measured in a second (allowing for decoherence, error correction etc).

I don't actually think you'll ever have to worry about that, though. There once was a computer with the following specification:

* 400 flops

* weighed over 27,000 kg

* big enough to park a school bus inside

* consumed 160 kw of power

* unreliable: it only worked half the time (literally!)

* programming it entailed rewiring it, which took days!

* price: $6 million in 2022 dollars

What a crap computer by today's standards! Absolute rubbish!

This was ENIAC in 1946. At the time, it was actually remarkably good - far better than anything that had gone before - and was really good for big calculations. Consequently, more computers were built, and they rapidly improved.

However, todays quantum computers are nowhere near as good as today's standard computers: not even cheap computers! So while they're a good idea, I don't think they're going to get the investment to develop like 1940s/1950s computers did.

[smatovic]

[...]
However, todays quantum computers are nowhere near as good as today's standard computers: not even cheap computers! So while they're a good idea, I don't think they're going to get the investment to develop like 1940s/1950s computers did.

Well, at some point Moore's Law is dead, we already have GPUs with 450W and CPUs with 300+W. 3nm, 2nm, 2+nm (I know, marketing, but reflects transistorcount/mm2) in pipe, but in the long-term we need alternatives, quantum, memristors, photonics, dunno.

--
Srdja

PS:

[towforce]

Well, at some point Moore's Law is dead, we already have GPUs with 450W and CPUs with 300+W. 3nm, 2nm, 2+nm (I know, marketing, but reflects transistorcount/mm2) in pipe, but in the long-term we need alternatives, quantum, memristors, photonics, dunno.

Not forgetting the topic of this thread - better software!

I know I'm in a minority here, but I am sincere in my beliefs that:

1. It's possible to play outstanding chess with a small and fast algorithm

2. While I obviously cannot prove this, I genuinely believe that the available evidence makes this more likely than not

PS:

[flok]

1. It's possible to play outstanding chess with a small and fast algorithm

like early versions of rofchade

[smatovic]

Not forgetting the topic of this thread - better software!

I know I'm in a minority here, but I am sincere in my beliefs that:

1. It's possible to play outstanding chess with a small and fast algorithm

2. While I obviously cannot prove this, I genuinely believe that the available evidence makes this more likely than not

I know

"God's Algorithm for Chess"
https://talkchess.com/forum3/viewtopic. ... 92#p932844

--
Srdja

[towforce]

1. It's possible to play outstanding chess with a small and fast algorithm

like early versions of rofchade

It sounds as though the early versions of Rofchade achieved a very good evaluation at very low cost (in "computer power"). However, when it switched to NNUE, two things happened:

1. "computer power" required for evaluation increased

2. The overall strength increased

So: it was a very good hand-crafted evaluation (HCE), but like all the other HCEs it missed some important things. We also know that super-fast deep searching also misses important things. It is very likely that all today's NNs also miss important things.

Based on what we know about NNs, it's extremely likely that an algorithm could be both smaller (hence faster) and more accurate. Given that "death by draw" has come to the top level of correspondence chess, a top quality small/fast chess algorithm would quite likely be able to get the best available result from most positions even on a cheap computer.

Anyway - back to the topic: if early versions of Rofchade get good results at low cost, then they would get a high logarithmic curve on my chart (earlier in this thread) of elo against computer power.