Engine programming

[Gejsi Marku]

Thank you all for some really good suggestions! I'll be bookmarking this for future reference

[Ovyron]

How good of a learner are you? I'm great at self-learning and have found success learning the most difficult things first, then the rest becomes easy, so in that case C++ and chess would be preferable to tic tac toe in an easy language, which you'd only do if you have difficulty self-learning.

[Gejsi Marku]

How good of a learner are you? I'm great at self-learning and have found success learning the most difficult things first, then the rest becomes easy, so in that case C++ and chess would be preferable to tic tac toe in an easy language, which you'd only do if you have difficulty self-learning.

Hi Ovyron, to date I have had no trouble learning things. I'm an Illustrator/Designer/Press operator by career. Like someone pointed out, for me is all about desire and fun. If I have those two learning becomes very easy for me (challenging yes, but that's where the fun is). Math and physics were my secret passion growing up, and later evolved into a strong interest in chess programming.
I'm probably not gonna be able to become competent enough anytime soon due to my other priorities, but thanks to many of your suggestions here I have a clearer path to where to focus my efforts.

[JohnWoe]

There's always room for more chess engines.
But I don't think there's more difficult first project than a chess engine.

I would start by writing a move generator and match PERFT numbers.

Chess has weird rules for en passant, castling etc. So implementing these will keep you busy for awhile. Then taking on UCI obscurities next...

As all computers are 64 bits. BITBOARDS makes a lot of sense. The whole board is presented by a simple number!
Magic bitboards could be added. So there's no loops for generating sliders.

Once it's generating moves *perfectly* add chess engine stuff ( NN / AB / Monte-Carlo / whatever ).

[TommyTC]

There is a YouTube series for the chess program "Vice":

Programming A Chess Engine in C
by Bluefever Software

[mvanthoor]

IMHO, there are a few suggestions:
1) Programming language: C++ is the best of the best for programming chess engines.

I disagree.

C and C++ have been in use for programming chess engines not because they are the 'best' languages, but because they are the fastest languages that aren't perfectly tight hand-coded assembler. There are better choices nowadays, IMHO. Here's why.

Even though information is much more widely available than even 15 years ago, let alone in the 90's or earlier, writing a chess engine from scratch is still not easy. C and C++ are languages with many pitfalls, and C++ is a very complex language with many features. It is very easy to do things the wrong way, or make small mistakes that'll bring the development of the engine to a grinding halt.

IMHO, if you want to write a chess engine from scratch and don't have a lot of programming experience, *use the language you know best*, as bob said. Even if this language is Java or even Python. Yes, the engine will be slow. Yes, it will never have a chance of becoming REALLY strong, due to lack of speed, but it offers the opportunity to concentrate on learning the chess engine concepts, without being bogged down by ALSO having to learn a new language. Write an engine first. Even if it's weak and slow, if it plays legal chess without errors and you can get it into a rating list, it can always be ported/rewritten into another, faster language. Then you can learn the language, without being bogged down by also learning the concepts of writing a chess engine.

If you don't even know a programming language, starting out with writing a chess engine might be a bridge too far... but you can still try. If you want or have to choose a programming language, my recommendation would be Rust. Not because I'm writing my own engine in Rust... but because of the language properties. I chose it because of:

- It's memory safe (as long as you don't use unsafe { } blocks.) No dangling pointers, ever. No NULL pointers.
- It's safe from deadlocks and race conditions. If you do it wrong, the compiler just WON'T compile your programs.
- rustc is a front-end to LLVM (*), just like clang is. (Clang / C-Lang, is a C-compiler). In essence, writing code in Rust and then compiling it with rustc, gives you (almost) the same machine code as writing it in C and compiling with clang. Therefore Rust (rustc) is just as fast as C (clang) or C++ (clang++). You don't have to worry about speed.
- Rust has an awesome language server (RLS, the Rust Language Server), which plugs directly into Visual Studio Code. This thing uses RustFMT (Rust Format) and clippy (the Rust Linter) to format and check your code. If you do _anything_ wrong, anywhere, RLS (and thus your editor) will warn you. before you even try to compile. If you want to change something somewhere, then just change it... and RLS and Clippy will point out where and how you need to change your code in other places.
- Adding a dependency is as easy as adding it to cargo.toml. (Just like package.json in NPM.)
- It doesn't have 500 gazilion compilers and build systems; it only has one: rustc as the compiler, cargo as the build system. (You can change this, but for a project written only in rust, there's no need.)
- Rust has no garbage collector, but it STILL does all of your memory management, using it's so called borrow checker. This borrow checker (which contains a very intelligent static code analyzer) warns you as soon as you try to do something that is not safe; like dropping a variable, and then later trying to use it again.

(*) LLVM is a compiler backend. It allows compilers in different languages to output their compiled code into LLVM intermittent code, after which LLVM optimizes it into machine code. This way, not every compiler needs to have its own machine code generator and optimizer.

Another option would be the Go language, but after comparing both of them extensively, I've chosen Rust, and up until now, I'm a very happy camper.

The only problem?

Rust seems A LOT harder than C and C++ when you're just starting out, because you're probably making A LOT of mistakes and / or doing unsafe things that C and C++ don't warn you about,but which get you into problems down the road. Rust warns you about everything, all the time, and only compiles your code when it's absolutely memory safe.

(It is not so that a Rust program can never crash / panic. That's very well possible if unexpected things happen. What can't happen, are all the subtle problems that are hard to find, such as memory leaks, double free for pointers, pointers that point nowhere, a switch statement that SHOULD switch between 5 options but only handles 4, and so on...)

[brianr]

I would like to provide another albeit isolated data point suggesting that the language speed does not matter at first, and for quite some time afterwards either. Testing my engine Tinker vs Stockfish and giving SF a 1:200 time handicap begins to make Tinker somewhat competitive. This was long before SF 11 or 10 or even 9, IIRC. So, the GUI development/debugging tool will be initially more important than the actual speed of nearly any code.

For your first engine the suggestion is to get comfortable writing and testing code and developing an engine that works and can play with a chess GUI. Then, take a look at the more advanced algorithms to produce a second generation stronger engine; but, even here, a good algorithm in search or evaluation is more important than raw speed.

By about your third generation engine speed will become a factor. Assuming you start with a "traditional" A/B engine, after your 3rd or 4th generation then if you look at the NN approach you will be astounded at how you can cook up a 3,000 Elo engine in a few days of training. Of course, it will be much more than a few days to understand how to train one, but you will more fully appreciate the beauty of the approach.

Finally, the "80/20 rule" will apply when trying to create a top tier very competitive engine (last bit of performance will be 80% of the effort). Very few can achieve that, but we can all enjoy the journey. Beware, chess programming is addictive.

[Alayan]

Speeding up an engine speeds up engine testing.

If A and B are functionally identical, but A runs at 2Mnps and B at 1Mnps, you can use half the TC when running patches for A and get similar games as you would have had with B.

When you're trying to make sure your engine isn't buggy, this may not be important, but as soon as you get in the elo race, speeding up patch testing is very, very, very nice. Hardware for patch testing and for tuning is a massive bottleneck for most engine authors.

However, making a first attempt, then starting again from scratch using the lessons learned in the first attempt could be a more efficient way than trying to get everything right the first time.

We see a similar principle with IID, where searching a shallow depth helps to make the later depths more efficient.

[Dann Corbit]

you get the same effect by multiplication of the hardware (2x hardware, 2x faster testing, 64 times as much hardware, 64 times faster testing {assuming the work is well coordinated}). This is the stockfish approach.

You can also get the same effect by multiplication of the speed of the hardware. E.g. testing with 128 core machine verses 8 cores.

Speeding up the program will also give a small Elo boost, assuming that the only functional change was speed.

[Ovyron]

Finally, the "80/20 rule" will apply when trying to create a top tier very competitive engine (last bit of performance will be 80% of the effort).

An important point here: Is this what he wants? The biggest problem I have seen is people thinking that computer chess programming is all about the elo the engine can produce, and measuring progress by it, so it becomes exclusively about finding ways to improving the engine's evaluation and search. Reinventing the wheel because people have already found solutions to those problems.

Engines like Rodent have shown that that's not necessarily the way to follow, while effort certainly was put into improving the engine's strength, I reckon it's the most used chess engine that is not at the top because its unique engine tweaking features and the way it allows to create unique personalities are pleasing an entire chunk of the population that don't really care about the elo the engine gets, but about what can it offer that allows the user to do something they couldn't do before.

If Gejsi Marku's goal is to have the engine being used by the most people possible and not just engine testers, perhaps he can focus on allowing users to do things we can't currently do, regardless of where the engine ends up in the rating lists.