Compare your engine's performance: Corrections +++
Moderator: Ras
-
- Posts: 24
- Joined: Tue Mar 16, 2021 11:11 pm
- Full name: Het Satasiya
Re: Compare your engine's performance: Corrections +++
NPS is a really garbage metric for measuring the strength of an engine. If you look at something like SF 7's nps vs A0's nps, SF7 was magnitudes faster in terms of nodes searched. But A0 was better than any engine out there, beating SF7 without a single loss. You don't even need engine writing experience to understand that, but it gives you serious context as to what is truly important.
-
- Posts: 313
- Joined: Tue Aug 03, 2021 2:41 pm
- Full name: Bill Beame
Re: Compare your engine's performance: Corrections +++
I also think there's a huge difference between writing a program to play games versus finding all mates. Playing games requires pruning. Pruning in mate finding loses accuracy. You don't have to be perfectly accurate to reach high ELO scores, you just have to be better than your opponent. The puzzle I initially listed has the following:amanjpro wrote: ↑Tue Aug 24, 2021 4:54 amklx wrote: ↑Tue Aug 24, 2021 3:30 amAgain, I did not mean to offend. Garbage was perhaps a strong word. Congrats on solving all those puzzles.Chessnut1071 wrote: ↑Tue Aug 24, 2021 3:09 am Question: If I give your 1 million nps engine 230 chess puzzles can it get all 230 solutions correct? By pruning specific types of moves, I can achieve much faster speeds too; however, it comes at a cost of accuracy. In fact, I can 2x the speed if I accept 92% accuracy. I started out at around 89,000 nps, but, as I included more puzzles, the speed dropped to around 23,500.
I have not written an engine, but I'd expect at least 1 million NPS single threaded. The total speed to solve a puzzle would come from two factors: time-per-node and number of nodes. I'd expect adding pruning to reduce the total node count, but also reduce the NPS since the pruning checks don't come for free. Have you made your code public? Happy to take a look at it and offer some suggestions. If your engine is alpha-beta based and your seeing 23,500 NPS there is guaranteed to be room for improvement, eg: excessive allocations (this is a common one).
I think you better write an engine before claiming these strong claims, or discovering your great ideas in the other threads...
My engine is 2730 CCRL and unless on very fast machines it cannot make to 1M NPS
So yeah, speak from experience next time
average moves: white = 45, black = 18
2-ply = 45 x 18 = 810
13- ply = (819)^7 = 2.28768E+20 possible moves
I use alpha-beta, Zobirst hash, and an evaluation using history, check and SEE. No pruning and I end up with 3,198,836,377 moves to find every mate in that puzzle. Even a 1 million nps engine will take 50 minutes to find all the mates. With strategic pruning you can get that down to a few minutes, however, that's specific to this particular puzzle and can't be extrapolated to others. My objective is to write a generalized perfect [near perfect] engine then speed it up later. I'm thinking about posting the FENs for those 230 puzzles-most taken from Classic Chess Puzzles K. Howard- so people can see where their engine fails.
-
- Posts: 179
- Joined: Tue Jun 15, 2021 8:11 pm
- Full name: Emanuel Torres
Re: Compare your engine's performance: Corrections +++
AlphaZero is a MCTS based method. OP's engine appears to be a standard "alpha-beta, Zobirst hash, and an evaluation using history, check and SEE". He's at 23,500 NPS and is asking for ways to increase speed. My bet is that a ~100x speedup is possible by optimizing the work done per node. What do you think?BrokenKeyboard wrote: ↑Tue Aug 24, 2021 6:04 am NPS is a really garbage metric for measuring the strength of an engine. If you look at something like SF 7's nps vs A0's nps, SF7 was magnitudes faster in terms of nodes searched. But A0 was better than any engine out there, beating SF7 without a single loss. You don't even need engine writing experience to understand that, but it gives you serious context as to what is truly important.
[Moderation warning] This signature violated the rule against commercial exhortations.
-
- Posts: 24
- Joined: Tue Mar 16, 2021 11:11 pm
- Full name: Het Satasiya
Re: Compare your engine's performance: Corrections +++
Whether its MCTS or not is kind of irrelevant here. And their goal is to first get their program as good as it can be before optimizing their code. While there might be optimizations possible, That isn't what they are trying to do.
-
- Posts: 179
- Joined: Tue Jun 15, 2021 8:11 pm
- Full name: Emanuel Torres
Re: Compare your engine's performance: Corrections +++
Huh? Throughout this thread OP has talked about optimizing his speed and he literally writes "Now, I'm focused on speed and would love to get some idea how he did it".BrokenKeyboard wrote: ↑Tue Aug 24, 2021 4:24 pm Whether its MCTS or not is kind of irrelevant here. And their goal is to first get their program as good as it can be before optimizing their code. While there might be optimizations possible, That isn't what they are trying to do.
And when he tells us that he's doing a "standard" minimax (not MCTS) and achieves 23,500 NPS, it's pretty obvious that there's huge potential there to improve the per-node speed.
[Moderation warning] This signature violated the rule against commercial exhortations.
-
- Posts: 24
- Joined: Tue Mar 16, 2021 11:11 pm
- Full name: Het Satasiya
Re: Compare your engine's performance: Corrections +++
Well hes looking for more algorithmical optimizations. And the "he" mentioned was JVM who made ChessMaster. chessnut was wondering how ChessMaster was so fast.
-
- Posts: 179
- Joined: Tue Jun 15, 2021 8:11 pm
- Full name: Emanuel Torres
Re: Compare your engine's performance: Corrections +++
Based on posts in this thread, ChessMaster is ~676x faster than OP's engine (55 sec vs 37200 sec). ChessMaster searches ~3M NPS. So 127x of the speed increase can be explained by faster speed per node, and only 5x by "algorithmic optimization" (visiting fewer nodes).BrokenKeyboard wrote: ↑Tue Aug 24, 2021 5:41 pm Well hes looking for more algorithmical optimizations. And the "he" mentioned was JVM who made ChessMaster. chessnut was wondering how ChessMaster was so fast.
[Moderation warning] This signature violated the rule against commercial exhortations.
-
- Posts: 338
- Joined: Sat Feb 25, 2012 10:42 pm
- Location: Stockholm
Re: Compare your engine's performance: Corrections +++
It would be fun to see your extremely fast original engine.klx wrote: ↑Tue Aug 24, 2021 5:05 pmHuh? Throughout this thread OP has talked about optimizing his speed and he literally writes "Now, I'm focused on speed and would love to get some idea how he did it".BrokenKeyboard wrote: ↑Tue Aug 24, 2021 4:24 pm Whether its MCTS or not is kind of irrelevant here. And their goal is to first get their program as good as it can be before optimizing their code. While there might be optimizations possible, That isn't what they are trying to do.
And when he tells us that he's doing a "standard" minimax (not MCTS) and achieves 23,500 NPS, it's pretty obvious that there's huge potential there to improve the per-node speed.
When you have a working engine some speed increase is quite easy to do. In C#, (at least if you have the professional version of Visual Studio) you can use the profiler under debug and choose cpu and record some positions to search to locate the functions that take time and see what statements in those functions take the time. In my engine I can see that the IsInCheck function is a big bottleneck and also that linq is bad for sorting.
Optimising the code to some degree is extremely easy. Writing the engine in first place is not
Doing some extremely smart things that h.g.m is doing to speed up capture search is not easy.
For me, I do not bother with speed so much since in the future my eval will be the huge bottleneck. One part of my eval will hopefully statically be able to evaluate some positions better than others will need a 10 plies search to do.
-
- Posts: 1404
- Joined: Wed Mar 08, 2006 10:15 pm
- Location: San Francisco, California
Re: Compare your engine's performance: Corrections +++
We may be using different versions of CM, as I am using CM9 - the one I'm most familiar with, and the one with the fewest bugs.klx wrote: ↑Tue Aug 24, 2021 6:32 pmBased on posts in this thread, ChessMaster is ~676x faster than OP's engine (55 sec vs 37200 sec). ChessMaster searches ~3M NPS. So 127x of the speed increase can be explained by faster speed per node, and only 5x by "algorithmic optimization" (visiting fewer nodes).BrokenKeyboard wrote: ↑Tue Aug 24, 2021 5:41 pm Well hes looking for more algorithmical optimizations. And the "he" mentioned was JVM who made ChessMaster. chessnut was wondering how ChessMaster was so fast.

-
- Posts: 179
- Joined: Tue Jun 15, 2021 8:11 pm
- Full name: Emanuel Torres
Re: Compare your engine's performance: Corrections +++
This sentence alone tells me so much. First, while C# is not terrible, I'd expect a ~2x slowdown over a language like C++. Second, linq is not something I'd ever want to see in performance critical code due to lack of control of allocations and work done. Third, sorting? For moves? You should not be doing move sorting at all, most of that will be wasted effort due to cutoffs. While performance is not a goal of yours I hope this illustrates the potential.
[Moderation warning] This signature violated the rule against commercial exhortations.