Progress on Rustic

[mvanthoor]

Sure, after enough puzzling and several rewrites, you may arrive at some point where the compiler doesn't complain. If you were ever to do that multi-threaded, you'd probably go for unsafe anyway because mutexing the access isn't an option.

It would be the same sort of puzzle, if I had written the engine in C++17. The conclusion is that my starting point was just extremely bad to begin with and I should have just written a new hash table to begin with. Second time around I just wrote a hash table with buckets that stored a single u8 as data (next to the zobrist verification and depth), replaced the u8 with a properly constrained generic, and it was done.

With regarding to multi-threading and unsafe hash table access: I've indeed read that if you go above 2 threads (some people say 4) that performance tanks if you use Mutex locking. That would indeed be one the (very few) things I need to do unsafe, but I'll look into that when I get there. Before the engine reaches 3000 Elo, I won't even have to touch multi-cpu/multi-threading, because engines such as Minic and Texel can reach 3100+ on a single CPU.

Regrettably, it seems you dislike Rust for the wrong reasons. Because you _sometimes_ may need unsafe, it doesn't mean that the entire language is worthless and could just have been unsafe because of it. Sometimes you need a piece of dynamite to get stuff done, but it doesn't mean that you have to try and light candles with it, because, neh... a match or a lighter is too safe. Real men light candles with dynamite. Personally I'd prefer a language that can be used safe for 98% of the time, having to drop to unsafe if really needed or wanted, as compared to a language that is unsafe all the time.

[lithander]

I checked out Weiss 1.3 on my Linux machine with AMD 3400G (older Zen+ architecture), and you have to edit the makefile a bit. The release versions with added "-DDEV" (for enabling perft) are at around 15 MNPS, except the PEXT one which is half of that, but that's expected due to my CPU with inefficient BMI2. The dev version itself is about 16 MNPS, i.e. the debug build isn't slow. The PGO (profile guided optimisation) build with "-DDEV" added is at 18 MNPS.

Considering the hardware difference (I'm using a AMD 3600 @4.2Ghz) the Cygwin seems to be just as performant as your Linux build. But the 15-20MNPS is only half of the 40 MNPS that Marcel remembered Weiss to generate moves at.

Since a 6700K should be about 6% faster than my 3400G in singlethread, you could download my engine to cross-check under Windows - perft is enabled also in the release EXE. Just remember to set the same position as Weiss.

I get 25MNPS with your engine on that position but weirdly I have to 'stop' the perft manually despite providing a depth argument.

I've rewritten the hash table from scratch. I can now initialize different hash tables by using different structs:

"let hash_table = HashTable::<PerftData>::new(64) " will create a 64 MB hash table for perft.
"let hash_table = HashTable::<SearchData>::new(64) " will create a 64 MB hash table for Search.

Wow, that was fast! Congratz!

One thing I noticed: when the bucket is an array of 4 elements, a 64 MB hash table is actually 64 MB; Rustic uses 64 MB + a bit of memory for itself. If I make the bucket a Vector with the same number of elements, the overhead becomes huge: a 64 MB hash table takes 106 MB of space. I wonder why.

Do you create your vector with a specific capacity? https://doc.rust-lang.org/std/vec/struc ... h_capacity

If not and you just let it grow than many container implementations (in C# also) just allocate some extra space so they can support additions and double the space at each reallocation. So the actual size only roughly correlates with the number of elements in the container and is usually ~1.5x bigger.

[mvanthoor]

Do you create your vector with a specific capacity? https://doc.rust-lang.org/std/vec/struc ... h_capacity

If not and you just let it grow than many container implementations (in C# also) just allocate some extra space so they can support additions and double the space at each reallocation. So the actual size only roughly correlates with the number of elements in the container and is usually ~1.5x bigger.

Yes. Now that you mention it, I remember reading somewhere that Rust actually does allocate 1.5 - 2x the amount of space requested, if you don't tell it otherwise. I'll have to look into that, because I don't want the program to allocate huge amounts of space it's never going to use.

The reason why I looked into a Vector is because then, I can use a variable number of entries per bucket. With a bucket as an array, the number of entries must be a constant. For the moment, that isn't really important because I can (still) fit 4 entries into one 64 kB bucket.

Wow, that was fast! Congratz!

As I said, I now learned how to do it. I also don't have a dynamic hash table anymore, and I just use a completely public data struct. That gets rid of the the generic interface for a generic table (which is a PITA to implement), and I don't need any (generic) setter/getter anymore. Now I just have a hash table with buckets, which hold entries, and each entry holds "verification: u32", "depth: u8", and "data: D". This makes the implementation much shorter and much more simple. Thus I can now just do:

let hash_table = HashTable::<some_data_struct>::new(<size_in_mb>);

I just have to see what's the best way to return stuff I need.

[Ras]

It would be the same sort of puzzle, if I had written the engine in C++17.

To be fair, I neither like C++. C hits a very sweet spot with more abstraction than assembly, and CPU portability, but not so much that abstraction would turn into obfuscation.

Considering the hardware difference (I'm using a AMD 3600 @4.2Ghz) the Cygwin seems to be just as performant as your Linux build. But the 15-20MNPS is only half of the 40 MNPS that Marcel remembered Weiss to generate moves at.

Maybe since then, Weiss has had further optimisations in move generation that are good in normal play, but slow down the move generator. Since the move generator doesn't take much time in regular search, anything that helps in generating moves with better sorting will easily outweigh the speed loss.

I fully agree with Marcel here that having a super fast perft is not a useful goal for an engine (perft tools are another story) if it doesn't mirror what's being done in actual search. Rustic won't keep up its perft speed either once stuff like depth killers and quiet move history get implemented, but it will gain strength nonetheless.

I get 25MNPS with your engine on that position but weirdly I have to 'stop' the perft manually despite providing a depth argument.

That's strange because it should stop automatically, although even perft can be interrupted with stop or quit. It does here when giving perft 5 as command, after about 10 seconds. More than 5 on Weiss' position will probably take very long, so I havn't tried that.

[mvanthoor]

Yesterday evening I re-implemented the generic hash table in Perft, and the command-line options to set the size. The only thing to do now is to put the UCI/XBoard options back in. I accidentally nuked them with the former branch. I forgot I had them merged in there already, without keeping their source branch.

This weekend I hope to get the time to actually test the addition of the hash table to the search. After I determine if/how much playing strength this results, I'll look into adding hash move ordering, and test again. Then Alpha 2 will finally be finished. (And then I'll can spend some time writing some more tutorial/book stuff.)

[ydebilloez]

Time control in Rustic alpha seems to be problematic if selecting st=5 in cutechess. (5 seconds per move).
Cutechess-cli is very strict and a few ms passed the allotted time causes a timeout.
Will this get fixed in next release or even in a patch release on alpha?

[mvanthoor]

Time control in Rustic alpha seems to be problematic if selecting st=5 in cutechess. (5 seconds per move).
Cutechess-cli is very strict and a few ms passed the allotted time causes a timeout.
Will this get fixed in next release or even in a patch release on alpha?

Thanks for the report. I've created an issue in the repository and I'll look into it for the next release. I'm not going to fix Alpha 1 and do a complete new release for this, because in the beginning, the releases will probably be fairly regular; about once a month or so, maybe 6 weeks. In the future, if a release can take months because of testing, I'll probably start doing version x.1 hotfixes.

I must admit that move time is not as extensively tested as game time (because I almost never use it for playing matches, because most testing lists also don't), and Rustic's time management is still fairly simplistic. That'll be incrementally improved over time

[lithander]

I'm not going to fix Alpha 1 and do a complete new release for this, because in the beginning, the releases will probably be fairly regular; about once a month or so, maybe 6 weeks. In the future, if a release can take months because of testing, I'll probably start doing version x.1 hotfixes.

It's not like Rustic Alpha is becoming obsolete when you release the next version with transposition tables etc because the new version will play much stronger. Some of us may like Rustic Alpha because it's current playing strength around 1600 is making it nice sparing partner whereas the next version would be just way too strong.

Though, personally I'm not concerned by the issue and just use incremental time controls. I've just reviewed my logs and never had an issue with Rustic losing because of timeout in that mode.

[ydebilloez]

Thanks for the report. I've created an issue in the repository and I'll look into it for the next release. I'm not going to fix Alpha 1 and do a complete new release for this, because in the beginning, the releases will probably be fairly regular; about once a month or so, maybe 6 weeks. In the future, if a release can take months because of testing, I'll probably start doing version x.1 hotfixes.

I understand, testing takes a lot of tine. Lots

I must admit that move time is not as extensively tested as game time (because I almost never use it for playing matches, because most testing lists also don't), and Rustic's time management is still fairly simplistic. That'll be incrementally improved over time

I like Rustic Alpha as it is way stronger as my engine, one some time controls at least. Why do I raise the issue?
Because I noticed that my upcoming version under certain time controls is about 400 elo stronger, while weaker on other time controls. So even if we only consider time controls as used on CCRL and others, it find it strange and an indicator of errors. (Pentium and Xeon scores are different as well !!!)

To give some examles Belofte 2.1.1 compared to 2.1.0 (partially tested on some levels)
# cutechess timecontrol (ref https://sourceforge.net/p/belofte/gitre ... version.sh)
"tc=40/25" -> 400 elo stronger
"tc=inf/60+0.6" -> 300 elo stronger
"tc=40/120+2" -> 100 elo lower
"st=5" -> 100 elo lower
"tc=inf depth=4" -> 150 elo lower
...

[mvanthoor]

I'm not going to fix Alpha 1 and do a complete new release for this, because in the beginning, the releases will probably be fairly regular; about once a month or so, maybe 6 weeks. In the future, if a release can take months because of testing, I'll probably start doing version x.1 hotfixes.

It's not like Rustic Alpha is becoming obsolete when you release the next version with transposition tables etc because the new version will play much stronger. Some of us may like Rustic Alpha because it's current playing strength around 1600 is making it nice sparing partner whereas the next version would be just way too strong.

Though, personally I'm not concerned by the issue and just use incremental time controls. I've just reviewed my logs and never had an issue with Rustic losing because of timeout in that mode.

Yeah, you're right. There isn't really a reason not to backport the fix to Alpha 1 and release Alpha 1.1 for people who want to use that version of the engine. I can imagine that I just haven't used any GUI overhead protection, as it normally isn't an issue with incremental time controls (except if your increment is lower than 0.1 seconds or thereabout).

I like Rustic Alpha as it is way stronger as my engine, one some time controls at least. Why do I raise the issue?
Because I noticed that my upcoming version under certain time controls is about 400 elo stronger, while weaker on other time controls. So even if we only consider time controls as used on CCRL and others, it find it strange and an indicator of errors. (Pentium and Xeon scores are different as well !!!)

To give some examles Belofte 2.1.1 compared to 2.1.0 (partially tested on some levels)
# cutechess timecontrol (ref https://sourceforge.net/p/belofte/gitre ... version.sh)
"tc=40/25" -> 400 elo stronger
"tc=inf/60+0.6" -> 300 elo stronger
"tc=40/120+2" -> 100 elo lower
"st=5" -> 100 elo lower
"tc=inf depth=4" -> 150 elo lower
...

I don't understand this post. Is it Belofte 2.1.1 against 2.1.0 that you are now comparing, so 2.1.1 is 400 Elo stronger than 2.1.0 on 40/25", or are you comparing against Rustic?

Belofte 2.1.1 isn't yet in your files download. I'm going to download version 2.1.0 and run some tests on my normal incremental time controls of 1m+0.6s (for quick testing if a change actually improves the engine), and 2m+1s (CCRL Blitz controls). For later versions, if Rustic gets into the 40/15 list, I may add tests at a comparable time control. (I'll have to ask how they determine the correct time control using Stockfish's benchmark.)

If Rustic is 400 Elo stronger than your engine on some time controls, and 100 Elo weaker at others, it is probably losing lots of games on time. If you have a better evaluation (more terms, probably) than Rustic, it can be the case that on fixed depth, your engine is actually stronger. When running on fixed depth, you're not comparing the complete engines; you're comparing the evaluations. I never use this for testing, because it's useless; Rustic only has material counting and PST's for now, so when tested against whatever engine that has a more extensive evaluation, it will lose; even if that engine is hundreds of points weaker in timed play.