@Discord guys: What is modern chess programming about?

[smatovic]

@Discord guys: What is modern chess programming about? What was old chess programming about?

Feel free to elaborate, my collection:

....well, maybe first define the timeframe what oldschool and newschool is, Discord members mention that CPW and TC are oldschool, and that "we" are missing the point of modern chess programming, so, I myself am interested, what I am missing in context of "what modern chess programming is about", or alike, what is modern chess programming about, and, what was old chess programming about?

--
Srdja

Actual statistical testing to get actual statistical signifficance is the most important part which wasn't the case prior to rybka more or less and why rybka was so dominant at it prime.
Ofc other things - SPSA, details of how to implement heuristics that gain - like LMR can be implemented to gain 5 elo or to gain 100 elo depending on implementation, but this is true for almost any known heuristic like null move pruning or futility pruning.

So,

- use SPRT for engine-engine self-play testing
- use SPSA for tuning of search heuristics parameters in LMR, null move pruning, futility pruning and alike

--
Srdja

Use SPSA also for eval, because you will have HCE.
Know details of implementation - how much to reduce moves in LMR (general look of formula, not -1-2-3), how much to reduce after a null move (the same), what margin to have in RFP, what heuristics are worth the most in hancrafted evaluation (aka what brings most elo to kingdanger, passers, etc) - this is extremely important if you are size/memory limited, you would want to implement from "big to small". Etc. Efficient way to do this was completely unknown in 1960 (at least this is what I assume or it was lost somewhere in the process of chess engines development to 2010).
Good basic logic + good tuning on top is literally worth hundreds of elo compared to what you can do "by feel". It's not even close.

The thing is yes. Most of techniques in principle were known long time ago/known in other black & white games.
Problem is simple - proper implementation of this techniques differs by literally thousand of elo to what it was 30 years ago. And let me tell you 1000 elo is somewhat of a big deal.

There are two groups: the people that understand the meaning behind the ICGA events and the people that don't (end users).
The event has never been to decide which is the best software for an end user.
It has always been an event to stimulate scientific research in the field. A computer scientist is far more than the average programmer.

The algorithms for parallelizing chess programs were invented and used in the ICGA event before the PCs ever had more than one processor.
Lots of things were developed at least in part for this event. It has been about what is the best possible chess playing entity combination of HW and SW.
There have been supercomputers, FPGA systems, grid systems ...

It is about pushing the envelope of the science! Thus it gives us unlimited imagination tob develop something new.
CCRL, CEGT, TCEC and the rest don't do that.
The reason d'etre of the ICGA events is completely different from that of the end user test groups and it is obvious that many end users don't come close to understanding that.

The reason why everyone sees it as only shitty tournament nowadays is because modern chess engine development isn't about science, it's about engineering. All innovations in past decades were not created/discussed at ICGA events, they were made by enthusiasts at fishtest and openbench.
And even if they were not - which is the case for such gamebreaking concepts as CNNs and NNUE, they were made not by any person at ICGA either, it's either deep mind and group of enthusiasts that reconstructed A0 logic, or shogi engine developers.
So in general "scientific relevance" of WCCC is basically 0 for more than a decade - Marco said it's irrelevant in as far as 2012 in this aspect, for example - so idk what "pushing envelope of the science" it does. It did in the past but this days are long time gone. You can't do this if you don't have literally anyone relevant participating - and with internet you don't even need them to group at one place.
TCEC as an event got multiple people interested in chess engine programming that eventually became accomplished devs, so it indeed did much better job at this in recent years, well, not even talking about being infinitely better tournament from any point of view.

--
Srdja

[smatovic]

E.g. neuronal nets are possible because fast graphic cards make progress.

You are aware that NNUE engines such as SF don't even use GPUs?

Well, they use one for training? Generate games, label games, train neural network, SPRT testing -> hardware, or better, meanwhile a cluster of hardware, both, Stockfish and Lc0. They say Vas started the race with big hardware, for tuning and testing, and Bob had his data center at university, to run 10K games to test 10 Elo and alike...

--
Srdja

[towforce]

The "Discord Guys" are not evil: if we're patient, they'll eventually start showing their good side.

[smatovic]

It is not about being good or evil, it is about engaging in an constructive discourse or not.

--
Srdja

[Viz]

Well, most important thing - SPRT everything, literally everything. And in self-play.
No 400, 800, 12000 games fixed games tests, not against pool of other engines (unless you want to test smth that is supposed to work only vs weaker or stronger engines like contempt but even for this you can just use older version of your engine or give it less/more time).
This is a pretty simple advice but one that will get you really really far.
Also don't try to fix smth based on one game or even 10 games because this literally never works. Almost never worked even for HCE (I can recall massive I think 2 times where patch was based on an actual game stockfish lost) but with NNUE this is borderline useless.

[Peter Berger]

It is not about being good or evil, it is about engaging in an constructive discourse or not.

--
Srdja

But it is.

Let's face it: CCC has never been the most civil of places, but at least the key persons involved wanted to keep it alive.

It is easy to break such a fragile place. Moderation is so clumsy here e.g., that no one in his right mind would want to do this more like once or twice a month.

It is nice, that some interaction is taking place right now, because of the potential to see a new generation of top developers taking part.
But of course, it is exactly about that: whether these guys want to be good guys +here+, or be part of the evil forces.

Just turn the situation around: suddenly everyone joined their discord channel and behaved badly. I am sure they'd manage to secure their home turf.

Peter

[smatovic]

Well, therefore my post and questions. What is modern chess programming about? What was old chess programming about? What are "we", here on TC, missing?

--
Srdja

[Viz]

Well, therefore my post and questions. What is modern chess programming about? What was old chess programming about? What are "we", here on TC, missing?

--
Srdja

Intuition is nothing, statistics are everything. What I mean - ofc intuition is needed, you need to write ideas that make sense and guess which of them are worth it, but everything needs to be statistically proven in a proper way by playing actual games.
Not 800 games, proper SPRT, not some metrics like time to depth, depth to time, average game length, branching factor and other stuff, just pure wins, draws and losses. Yes, it's annoying and requires a lot of computation, but this is the only real way to progress. And if you don't have enough resources - lowering time controls that you test on is infinitely better than committing something on insufficient amount of games, no matter how long this games are. This also directly leads that you can't realistically test almost anything on time controls longer than bullet, unless you have some things that you have big suspicion to scale non-linearly.
Precise numbers are also found with automated tuning - but then need to also be statistically tested. SPRT is the king, everything else is almost always completely useless.
Old chess programming is basically not doing this.
Using some bogus metrics that mean nothing, playing 800 games "but at least this is rapid", guessing values based on god only knows what basis, using other engines instead of selfplay (this can be done but will increase computational time required for any patch by some 2x factor or so).

[smatovic]

Intuition is nothing, statistics are everything.
[...]

And here I disagree. As far as I got it there are no test-suites used as regression test on Fishtest. Why? Cos it does not give Elo? The aim of Stockfish is to be the strongest engine, Elo wise? But there are other engines, with different objectives, Elo, "statistics", is just one possible metric in computer chess.

--
Srdja

[chrisw]

Well, therefore my post and questions. What is modern chess programming about? What was old chess programming about? What are "we", here on TC, missing?

--
Srdja

Intuition is nothing, statistics are everything. What I mean - ofc intuition is needed, you need to write ideas that make sense and guess which of them are worth it, but everything needs to be statistically proven in a proper way by playing actual games.
Not 800 games, proper SPRT, not some metrics like time to depth, depth to time, average game length, branching factor and other stuff, just pure wins, draws and losses. Yes, it's annoying and requires a lot of computation, but this is the only real way to progress. And if you don't have enough resources - lowering time controls that you test on is infinitely better than committing something on insufficient amount of games, no matter how long this games are. This also directly leads that you can't realistically test almost anything on time controls longer than bullet, unless you have some things that you have big suspicion to scale non-linearly.
Precise numbers are also found with automated tuning - but then need to also be statistically tested. SPRT is the king, everything else is almost always completely useless.
Old chess programming is basically not doing this.
Using some bogus metrics that mean nothing, playing 800 games "but at least this is rapid", guessing values based on god only knows what basis, using other engines instead of selfplay (this can be done but will increase computational time required for any patch by some 2x factor or so).

So, it’s wheels, gears, structure, electrical systems already invented, the work already done, to be tuned better? No wonder we see a very low cost of entry and a gazillion new “engine developers”. Some of whom have the audacity to call themselves “top engine developers”. Takes what to not reinvent any wheels? A few months? Or weeks?