Progress on Blunder

[j.t.]

I am looking at something that is as cheap as possible, with as many cores as possible.

You could look on eBay for old servers with dual sockets. They often have 24 threads with reasonably fast CPUs, for example I found this: Lenovo ThinkServer TD340 (24 threads, 350 €) or Supermicro Server CSE-826 (32 threads, 400 €). Would surely be better than 30 Raspberry Pies. Of course, they might have issues as they were probably under heavy usage. I got a case where the fans didn't throttle, and server fans under high power are not really quiet ...

[mvanthoor]

You could look on eBay for old servers with dual sockets. They often have 24 threads with reasonably fast CPUs, for example I found this: Lenovo ThinkServer TD340 (24 threads, 350 €) or Supermicro Server CSE-826 (32 threads, 400 €). Would surely be better than 30 Raspberry Pies. Of course, they might have issues as they were probably under heavy usage. I got a case where the fans didn't throttle, and server fans under high power are not really quiet ...

They will probably also burn through my bank account really quickly because of power usage... And, when using much newer CPU's, the time controls can probably be MUCH shorter to reach the same depths, which also makes testing faster. Maybe the 8-core/16-thread 5700G could run 12 games concurrently. That would be 3x more games concurrently than I can now run on my 6700K. It would cut testing time to 1/3rd.

[jdart]

You could look on eBay for old servers with dual sockets.

I have several of these. Many companies depreciate hardware to zero in three years. So you can find 3-year-old equipment at pretty reasonable prices.

[Rebel]

You could look on eBay for old servers with dual sockets. They often have 24 threads with reasonably fast CPUs, for example I found this: Lenovo ThinkServer TD340 (24 threads, 350 €) or Supermicro Server CSE-826 (32 threads, 400 €). Would surely be better than 30 Raspberry Pies. Of course, they might have issues as they were probably under heavy usage. I got a case where the fans didn't throttle, and server fans under high power are not really quiet ...

They will probably also burn through my bank account really quickly because of power usage... And, when using much newer CPU's, the time controls can probably be MUCH shorter to reach the same depths, which also makes testing faster. Maybe the 8-core/16-thread 5700G could run 12 games concurrently. That would be 3x more games concurrently than I can now run on my 6700K. It would cut testing time to 1/3rd.

I get all my stuff from here - https://www.queensystems.nl/

Review - http://rebel13.nl/home/hardware.html

[algerbrex]

I also think an engine that learns to play chess is more interesting (technically) than one that has a lot of human chess knowledge written into an HCE. And NNUE has proven this approach to be valid. But also just tuning PSTs from a set of annotated games is a form of machine learning. And when I trained my tables I started with just the basic material values that every human child learns together with the rules. The nuances, how to play an opening and win an endgame, all that was something the engine learned "itself" and not from me.

Yeah, I also think that's one of the more interesting parts of writing an engine, and I'll definitely be exploring NNUE in Blunder in the coming months. But I'm also curious to see how far I can stretch HCE using Texel tuning. There are so many variables with Texel tuning, trying to find its limits has always been something that's interested me.

And I think there's a huge blank space between simple PSTs and NNUE...

That's definitely true. I originally didn't plan on adding any terms besides PSTs for a while, but PST really can only get you so far. And it got pretty old seeing Blunder double, and even sometimes triple its pawns, only to lose them all in the endgame.

I have also written some handcrafted evaluation terms but never commited them to the master branch even though they added strength. And so with Leorik I plan to climb the Elo ladder with machine learning techniques but step by step. I imagine to explore the whole space between PST and NNUE.

That sounds a little bit like the point I'm at. Testing the limits of things. Hopefully we'll both be able to share what we learn.

And maybe I'll try some fun distractions too. For example imagine a simulated evolution where you mutate the PSTs and let the engines that win procreate!

I actually tried that for a little bit a couple of months ago. Not exactly like what you described, but I came across an interesting paper that claimed to use genetic learning to tune the search parameters of a chess engine with positive results. Never worked out for me though, most likely because I failed to implement the algorithm correctly.

I think right now I've made up my mind to re-write Blunder. I've always had a problem with starting over with a project too many times, but I feel like I can get a positive result, just like what happened between Blunder 4.0.0 and Blunder 5.0.0. Whenever I finish the re-write, I'll likely release it as Blunder 8.0.0 since at the very minimum, it'll be an extensive modification to the code-base. Hopefully, that'll also come with a good bit of Elo gain as well.

[algerbrex]

That's a lot of games.... is my math really off when I count the things I still need to implement?

- Rustic 4 (Tapered Evaluation): 2160
- Rustic 5 (Null move pruning + static null move pruning): + 140 => 2300
- Rustic 6 (Three-stage move generatiion) +40 => 2340
- Rustic 7 (Aspiration Window + History) +40 (? going by other engines) => 2380
- Rustic 8 (LMR) +100 (? going by other engines) => 2480

Those numbers look about right to me, and with the speed of rustic, I think some of them could be even a bit more. You may have to play with the static null move pruning a bit. I've found it's pretty sensitive to the margin used. Feel free to start with numbers from Blunder and tweak from there as needed.

A lot of the feature I've tested that have needed a couple thousands games to verify have been more minor search patches.

For example, making sure LMR doesn't drop you directly into qsearch gained Blunder 21 Elo. But that only became clear after about 6000+ games, at 10+0.1s. Other things like evaluation patches have also taken more games than typical features.

Then there are some things such as futility pruning and SEE that I haven't even thought about. There's no pawn hash, pawn structure evaluation, king safety, or whatever... no evaluation yet on top of the PST's. Still I expect Rustic to be able to hit 2400 - 2450 at least, without even touching the evaluation.

You may be able to, and I'd like to see you get there. I remember the biggest ceiling for Blunder to crack 2500 was king safety. It became more and more obvious the more games Blunder played against engines with even minimal king safety. They'd crack Blunder's king wide open and it'd be over in the next 10 moves. YMMV though.

But, maybe my math is completely off, because most other engines develop both the search and the evaluation in tandem and the mentioned Elo ratings for those engines aren't even close for mine. We'll see.

Eh, I'm sure you know this already, but I've found once you start moving away from some of the super mainstream features, things become really engine-dependent. I remember a lot of people saying static null move pruning wasn't worth much at all, but I remember getting a good bit of Elo from it, something like 60-ish Elo. So I think the best thing is just to use other people's experiences as a rough rule of thumb and work from there.

What do you have in mind for this testing cluster? I am looking at something that is as cheap as possible, with as many cores as possible. Because I've moved my DGT board to a NUC, I have two RPi4's lying around. Maybe I could build an 8x 4Pi testing cluster somehow, to test with 32 cores, but the Pi 4 gets really hot. If not, I'll go ahead and build a test computer on the basis of an AMD Ryzen 5700G (8 cores) into a mini-ITX case as a dedicated testing rig. (I'd not recommend an 8-core ASUS or Gigabyte NUC-like computer for this, because those systems have fans that scream your head off when the CPU gets hot. Better build a somewhat bigger system using the biggest cooler and fan you can fit into a case as small as possible.)

Tbh, I haven't given this a lot of thought over this past month. My original idea was to purchase a bunch of raspberry pi's and build a cluster from them, something like you're describing. I wasn't aware of the heating issues though, so that'll be something I'll need to look into before committing that way.

[algerbrex]

I am looking at something that is as cheap as possible, with as many cores as possible.

You could look on eBay for old servers with dual sockets. They often have 24 threads with reasonably fast CPUs, for example I found this: Lenovo ThinkServer TD340 (24 threads, 350 €) or Supermicro Server CSE-826 (32 threads, 400 €). Would surely be better than 30 Raspberry Pies. Of course, they might have issues as they were probably under heavy usage. I got a case where the fans didn't throttle, and server fans under high power are not really quiet ...

I'll check those out too thanks. Rasberry pi's were my original plan, but I think going with building a dedicated multi-core computer for a testing cluster will be more durable and stable in the long run, even though it'll likely be more expensive.

[mvanthoor]

Tbh, I haven't given this a lot of thought over this past month. My original idea was to purchase a bunch of raspberry pi's and build a cluster from them, something like you're describing. I wasn't aware of the heating issues though, so that'll be something I'll need to look into before committing that way.

I have two Pi4's that I use(d) for PicoChess and compiling Rustic for the Pi. I've also run some tests, obviously. It really got too hot to handle, with temperatures in the range of 75 degrees. With my Pi's i use the Argon ONE case, which also acts as a heat sink, and it got hot enough that it couldn't be picked up and held anymore if the Pi was running at 100% CPU power.

Because of that I'm hesitant to build a cluster on the basis of the Pi4. I would never leave a stack of 10 of them running unattended for hours.

[j.t.]

I'll check those out too thanks. Rasberry pi's were my original plan, but I think going with building a dedicated multi-core computer for a testing cluster will be more durable and stable in the long run, even though it'll likely be more expensive.

Keep in mind this note by mvanthoor:

They will probably also burn through my bank account really quickly because of power usage... And, when using much newer CPU's, the time controls can probably be MUCH shorter to reach the same depths, which also makes testing faster. Maybe the 8-core/16-thread 5700G could run 12 games concurrently. That would be 3x more games concurrently than I can now run on my 6700K. It would cut testing time to 1/3rd.

If the 5700G is twice as fast as the CPUs you would get in a used server, then possibly this is really a better investment. Though actually knowing how fast multicore usage of a 5700G vs an old server Xeon is, is maybe not quite easy, as the 5700G likely will throttle more when all cores are running than the Xeon. In the end, you want to optimize computing-power/cost (probably under the constraint that you want at least X computing-power). I would guess that a server CPU or a 5700G both will be less expensive than such a number of raspberry pies, that they are equal in performance.

Also, a 5700G is probably more useful when you don't want to use it for chess development anymore than an old server CPU or a raspi cluster. Though I have to admit, a raspi cluster is more interesting (especially visually)

[mvanthoor]

...

Why would the 5700G throttle? I don't expect it to throttle at all. I have a fairly normal cooler on top of an older 6700K, which is a 95W CPU. The 5700G is a CPU with a 45-65W TDP; so it runs cooler than the 6700K. However, it is three times as fast.

I now run my tests at 10s+0.1 on the 6700K; the deeper the engine can search in a given time, the better the result will be. Thus on a very old CPU I would need to extend that testing time. I don't see myself running 10s+0.1s on a Raspberry Pi, because the results would be almost meaningless, as the engine can't get deep enough. (On the 6700K, with its current feature-set, my engine searches 6-10 moves deep depending on the position.)

On the much... MUCH faster 5700G, can probably run 12 games concurrently (I don't like to use all of the SMT/HyperThreading threads for chess engines), and maybe even decrease the testing time from 10s+0.1 to 8s+0.08 (but I doubt I'll be doing that).

Building a small system around the 5700G will cost around €750 though; of which €340 is for the CPU alone. (And a fairly high-end mainboard because it will be running A LOT.)

If all goes well, I'll be building that system after the release of Rustic 4.