Progress on Rustic

Discussion of chess software programming and technical issues.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
mvanthoor
Posts: 740
Joined: Wed Jul 03, 2019 2:42 pm
Full name: Marcel Vanthoor

Re: Progress on Rustic

Post by mvanthoor » Tue Feb 23, 2021 12:35 am

Ras wrote:
Mon Feb 22, 2021 11:14 pm
Sure, after enough puzzling and several rewrites, you may arrive at some point where the compiler doesn't complain. If you were ever to do that multi-threaded, you'd probably go for unsafe anyway because mutexing the access isn't an option.
It would be the same sort of puzzle, if I had written the engine in C++17. The conclusion is that my starting point was just extremely bad to begin with and I should have just written a new hash table to begin with. Second time around I just wrote a hash table with buckets that stored a single u8 as data (next to the zobrist verification and depth), replaced the u8 with a properly constrained generic, and it was done.

With regarding to multi-threading and unsafe hash table access: I've indeed read that if you go above 2 threads (some people say 4) that performance tanks if you use Mutex locking. That would indeed be one the (very few) things I need to do unsafe, but I'll look into that when I get there. Before the engine reaches 3000 Elo, I won't even have to touch multi-cpu/multi-threading, because engines such as Minic and Texel can reach 3100+ on a single CPU.

Regrettably, it seems you dislike Rust for the wrong reasons. Because you _sometimes_ may need unsafe, it doesn't mean that the entire language is worthless and could just have been unsafe because of it. Sometimes you need a piece of dynamite to get stuff done, but it doesn't mean that you have to try and light candles with it, because, neh... a match or a lighter is too safe. Real men light candles with dynamite. Personally I'd prefer a language that can be used safe for 98% of the time, having to drop to unsafe if really needed or wanted, as compared to a language that is unsafe all the time.
Author of Rustic.
Releases | Code | Docs

User avatar
lithander
Posts: 65
Joined: Sun Dec 27, 2020 1:40 am
Location: Bremen, Germany
Full name: Thomas Jahn

Re: Progress on Rustic

Post by lithander » Tue Feb 23, 2021 10:43 am

Ras wrote:
Mon Feb 22, 2021 10:45 pm
I checked out Weiss 1.3 on my Linux machine with AMD 3400G (older Zen+ architecture), and you have to edit the makefile a bit. The release versions with added "-DDEV" (for enabling perft) are at around 15 MNPS, except the PEXT one which is half of that, but that's expected due to my CPU with inefficient BMI2. The dev version itself is about 16 MNPS, i.e. the debug build isn't slow. The PGO (profile guided optimisation) build with "-DDEV" added is at 18 MNPS.
Considering the hardware difference (I'm using a AMD 3600 @4.2Ghz) the Cygwin seems to be just as performant as your Linux build. But the 15-20MNPS is only half of the 40 MNPS that Marcel remembered Weiss to generate moves at.
Ras wrote:
Mon Feb 22, 2021 10:45 pm
Since a 6700K should be about 6% faster than my 3400G in singlethread, you could download my engine to cross-check under Windows - perft is enabled also in the release EXE. Just remember to set the same position as Weiss.
I get 25MNPS with your engine on that position but weirdly I have to 'stop' the perft manually despite providing a depth argument.
mvanthoor wrote:
Mon Feb 22, 2021 10:24 pm
I've rewritten the hash table from scratch. I can now initialize different hash tables by using different structs:

"let hash_table = HashTable::<PerftData>::new(64) " will create a 64 MB hash table for perft.
"let hash_table = HashTable::<SearchData>::new(64) " will create a 64 MB hash table for Search.
Wow, that was fast! Congratz! :)
mvanthoor wrote:
Mon Feb 22, 2021 10:24 pm
One thing I noticed: when the bucket is an array of 4 elements, a 64 MB hash table is actually 64 MB; Rustic uses 64 MB + a bit of memory for itself. If I make the bucket a Vector with the same number of elements, the overhead becomes huge: a 64 MB hash table takes 106 MB of space. I wonder why.
Do you create your vector with a specific capacity? https://doc.rust-lang.org/std/vec/struc ... h_capacity

If not and you just let it grow than many container implementations (in C# also) just allocate some extra space so they can support additions and double the space at each reallocation. So the actual size only roughly correlates with the number of elements in the container and is usually ~1.5x bigger.
Minimal Chess. My very first chess engine! Details on Youtube & Github

User avatar
mvanthoor
Posts: 740
Joined: Wed Jul 03, 2019 2:42 pm
Full name: Marcel Vanthoor

Re: Progress on Rustic

Post by mvanthoor » Tue Feb 23, 2021 11:17 am

lithander wrote:
Tue Feb 23, 2021 10:43 am
Do you create your vector with a specific capacity? https://doc.rust-lang.org/std/vec/struc ... h_capacity

If not and you just let it grow than many container implementations (in C# also) just allocate some extra space so they can support additions and double the space at each reallocation. So the actual size only roughly correlates with the number of elements in the container and is usually ~1.5x bigger.
Yes. Now that you mention it, I remember reading somewhere that Rust actually does allocate 1.5 - 2x the amount of space requested, if you don't tell it otherwise. I'll have to look into that, because I don't want the program to allocate huge amounts of space it's never going to use.

The reason why I looked into a Vector is because then, I can use a variable number of entries per bucket. With a bucket as an array, the number of entries must be a constant. For the moment, that isn't really important because I can (still) fit 4 entries into one 64 kB bucket.
lithander wrote:
Tue Feb 23, 2021 10:43 am
Wow, that was fast! Congratz! :)
As I said, I now learned how to do it. I also don't have a dynamic hash table anymore, and I just use a completely public data struct. That gets rid of the the generic interface for a generic table (which is a PITA to implement), and I don't need any (generic) setter/getter anymore. Now I just have a hash table with buckets, which hold entries, and each entry holds "verification: u32", "depth: u8", and "data: D". This makes the implementation much shorter and much more simple. Thus I can now just do:

let hash_table = HashTable::<some_data_struct>::new(<size_in_mb>);

I just have to see what's the best way to return stuff I need.
Author of Rustic.
Releases | Code | Docs

Ras
Posts: 1745
Joined: Tue Aug 30, 2016 6:19 pm
Full name: Rasmus Althoff
Contact:

Re: Progress on Rustic

Post by Ras » Tue Feb 23, 2021 3:06 pm

mvanthoor wrote:
Tue Feb 23, 2021 12:35 am
It would be the same sort of puzzle, if I had written the engine in C++17.
To be fair, I neither like C++. C hits a very sweet spot with more abstraction than assembly, and CPU portability, but not so much that abstraction would turn into obfuscation.
lithander wrote:
Tue Feb 23, 2021 10:43 am
Considering the hardware difference (I'm using a AMD 3600 @4.2Ghz) the Cygwin seems to be just as performant as your Linux build. But the 15-20MNPS is only half of the 40 MNPS that Marcel remembered Weiss to generate moves at.
Maybe since then, Weiss has had further optimisations in move generation that are good in normal play, but slow down the move generator. Since the move generator doesn't take much time in regular search, anything that helps in generating moves with better sorting will easily outweigh the speed loss.

I fully agree with Marcel here that having a super fast perft is not a useful goal for an engine (perft tools are another story) if it doesn't mirror what's being done in actual search. Rustic won't keep up its perft speed either once stuff like depth killers and quiet move history get implemented, but it will gain strength nonetheless.
I get 25MNPS with your engine on that position but weirdly I have to 'stop' the perft manually despite providing a depth argument.
That's strange because it should stop automatically, although even perft can be interrupted with stop or quit. It does here when giving perft 5 as command, after about 10 seconds. More than 5 on Weiss' position will probably take very long, so I havn't tried that.
Rasmus Althoff
https://www.ct800.net

Post Reply