Compare your engine's performance: Corrections +++

Discussion of chess software programming and technical issues.

Moderator: Ras

BrokenKeyboard
Posts: 24
Joined: Tue Mar 16, 2021 11:11 pm
Full name: Het Satasiya

Re: Compare your engine's performance: Corrections +++

Post by BrokenKeyboard »

NPS is a really garbage metric for measuring the strength of an engine. If you look at something like SF 7's nps vs A0's nps, SF7 was magnitudes faster in terms of nodes searched. But A0 was better than any engine out there, beating SF7 without a single loss. You don't even need engine writing experience to understand that, but it gives you serious context as to what is truly important.
Chessnut1071
Posts: 313
Joined: Tue Aug 03, 2021 2:41 pm
Full name: Bill Beame

Re: Compare your engine's performance: Corrections +++

Post by Chessnut1071 »

amanjpro wrote: Tue Aug 24, 2021 4:54 am
klx wrote: Tue Aug 24, 2021 3:30 am
Chessnut1071 wrote: Tue Aug 24, 2021 3:09 am Question: If I give your 1 million nps engine 230 chess puzzles can it get all 230 solutions correct? By pruning specific types of moves, I can achieve much faster speeds too; however, it comes at a cost of accuracy. In fact, I can 2x the speed if I accept 92% accuracy. I started out at around 89,000 nps, but, as I included more puzzles, the speed dropped to around 23,500.
Again, I did not mean to offend. Garbage was perhaps a strong word. Congrats on solving all those puzzles.

I have not written an engine, but I'd expect at least 1 million NPS single threaded. The total speed to solve a puzzle would come from two factors: time-per-node and number of nodes. I'd expect adding pruning to reduce the total node count, but also reduce the NPS since the pruning checks don't come for free. Have you made your code public? Happy to take a look at it and offer some suggestions. If your engine is alpha-beta based and your seeing 23,500 NPS there is guaranteed to be room for improvement, eg: excessive allocations (this is a common one).

I think you better write an engine before claiming these strong claims, or discovering your great ideas in the other threads...

My engine is 2730 CCRL and unless on very fast machines it cannot make to 1M NPS

So yeah, speak from experience next time
I also think there's a huge difference between writing a program to play games versus finding all mates. Playing games requires pruning. Pruning in mate finding loses accuracy. You don't have to be perfectly accurate to reach high ELO scores, you just have to be better than your opponent. The puzzle I initially listed has the following:

average moves: white = 45, black = 18
2-ply = 45 x 18 = 810
13- ply = (819)^7 = 2.28768E+20 possible moves

I use alpha-beta, Zobirst hash, and an evaluation using history, check and SEE. No pruning and I end up with 3,198,836,377 moves to find every mate in that puzzle. Even a 1 million nps engine will take 50 minutes to find all the mates. With strategic pruning you can get that down to a few minutes, however, that's specific to this particular puzzle and can't be extrapolated to others. My objective is to write a generalized perfect [near perfect] engine then speed it up later. I'm thinking about posting the FENs for those 230 puzzles-most taken from Classic Chess Puzzles K. Howard- so people can see where their engine fails.
klx
Posts: 179
Joined: Tue Jun 15, 2021 8:11 pm
Full name: Emanuel Torres

Re: Compare your engine's performance: Corrections +++

Post by klx »

BrokenKeyboard wrote: Tue Aug 24, 2021 6:04 am NPS is a really garbage metric for measuring the strength of an engine. If you look at something like SF 7's nps vs A0's nps, SF7 was magnitudes faster in terms of nodes searched. But A0 was better than any engine out there, beating SF7 without a single loss. You don't even need engine writing experience to understand that, but it gives you serious context as to what is truly important.
AlphaZero is a MCTS based method. OP's engine appears to be a standard "alpha-beta, Zobirst hash, and an evaluation using history, check and SEE". He's at 23,500 NPS and is asking for ways to increase speed. My bet is that a ~100x speedup is possible by optimizing the work done per node. What do you think?
[Moderation warning] This signature violated the rule against commercial exhortations.
BrokenKeyboard
Posts: 24
Joined: Tue Mar 16, 2021 11:11 pm
Full name: Het Satasiya

Re: Compare your engine's performance: Corrections +++

Post by BrokenKeyboard »

Whether its MCTS or not is kind of irrelevant here. And their goal is to first get their program as good as it can be before optimizing their code. While there might be optimizations possible, That isn't what they are trying to do.
klx
Posts: 179
Joined: Tue Jun 15, 2021 8:11 pm
Full name: Emanuel Torres

Re: Compare your engine's performance: Corrections +++

Post by klx »

BrokenKeyboard wrote: Tue Aug 24, 2021 4:24 pm Whether its MCTS or not is kind of irrelevant here. And their goal is to first get their program as good as it can be before optimizing their code. While there might be optimizations possible, That isn't what they are trying to do.
Huh? Throughout this thread OP has talked about optimizing his speed and he literally writes "Now, I'm focused on speed and would love to get some idea how he did it".

And when he tells us that he's doing a "standard" minimax (not MCTS) and achieves 23,500 NPS, it's pretty obvious that there's huge potential there to improve the per-node speed.
[Moderation warning] This signature violated the rule against commercial exhortations.
BrokenKeyboard
Posts: 24
Joined: Tue Mar 16, 2021 11:11 pm
Full name: Het Satasiya

Re: Compare your engine's performance: Corrections +++

Post by BrokenKeyboard »

Well hes looking for more algorithmical optimizations. And the "he" mentioned was JVM who made ChessMaster. chessnut was wondering how ChessMaster was so fast.
klx
Posts: 179
Joined: Tue Jun 15, 2021 8:11 pm
Full name: Emanuel Torres

Re: Compare your engine's performance: Corrections +++

Post by klx »

BrokenKeyboard wrote: Tue Aug 24, 2021 5:41 pm Well hes looking for more algorithmical optimizations. And the "he" mentioned was JVM who made ChessMaster. chessnut was wondering how ChessMaster was so fast.
Based on posts in this thread, ChessMaster is ~676x faster than OP's engine (55 sec vs 37200 sec). ChessMaster searches ~3M NPS. So 127x of the speed increase can be explained by faster speed per node, and only 5x by "algorithmic optimization" (visiting fewer nodes).
[Moderation warning] This signature violated the rule against commercial exhortations.
Pio
Posts: 338
Joined: Sat Feb 25, 2012 10:42 pm
Location: Stockholm

Re: Compare your engine's performance: Corrections +++

Post by Pio »

klx wrote: Tue Aug 24, 2021 5:05 pm
BrokenKeyboard wrote: Tue Aug 24, 2021 4:24 pm Whether its MCTS or not is kind of irrelevant here. And their goal is to first get their program as good as it can be before optimizing their code. While there might be optimizations possible, That isn't what they are trying to do.
Huh? Throughout this thread OP has talked about optimizing his speed and he literally writes "Now, I'm focused on speed and would love to get some idea how he did it".

And when he tells us that he's doing a "standard" minimax (not MCTS) and achieves 23,500 NPS, it's pretty obvious that there's huge potential there to improve the per-node speed.
It would be fun to see your extremely fast original engine.

When you have a working engine some speed increase is quite easy to do. In C#, (at least if you have the professional version of Visual Studio) you can use the profiler under debug and choose cpu and record some positions to search to locate the functions that take time and see what statements in those functions take the time. In my engine I can see that the IsInCheck function is a big bottleneck and also that linq is bad for sorting.

Optimising the code to some degree is extremely easy. Writing the engine in first place is not

Doing some extremely smart things that h.g.m is doing to speed up capture search is not easy.

For me, I do not bother with speed so much since in the future my eval will be the huge bottleneck. One part of my eval will hopefully statically be able to evaluate some positions better than others will need a 10 plies search to do.
JVMerlino
Posts: 1404
Joined: Wed Mar 08, 2006 10:15 pm
Location: San Francisco, California

Re: Compare your engine's performance: Corrections +++

Post by JVMerlino »

klx wrote: Tue Aug 24, 2021 6:32 pm
BrokenKeyboard wrote: Tue Aug 24, 2021 5:41 pm Well hes looking for more algorithmical optimizations. And the "he" mentioned was JVM who made ChessMaster. chessnut was wondering how ChessMaster was so fast.
Based on posts in this thread, ChessMaster is ~676x faster than OP's engine (55 sec vs 37200 sec). ChessMaster searches ~3M NPS. So 127x of the speed increase can be explained by faster speed per node, and only 5x by "algorithmic optimization" (visiting fewer nodes).
We may be using different versions of CM, as I am using CM9 - the one I'm most familiar with, and the one with the fewest bugs. :wink: On my pretty decent hardware, this single-threaded version of The King gets about 1M NPS from the initial position, and about 2.4M NPS in the OP's position. But, of course, back when this was released in 2002, we were happy to hit 100K NPS.
klx
Posts: 179
Joined: Tue Jun 15, 2021 8:11 pm
Full name: Emanuel Torres

Re: Compare your engine's performance: Corrections +++

Post by klx »

Pio wrote: Tue Aug 24, 2021 6:47 pmlinq is bad for sorting
This sentence alone tells me so much. First, while C# is not terrible, I'd expect a ~2x slowdown over a language like C++. Second, linq is not something I'd ever want to see in performance critical code due to lack of control of allocations and work done. Third, sorting? For moves? You should not be doing move sorting at all, most of that will be wasted effort due to cutoffs. While performance is not a goal of yours I hope this illustrates the potential.
[Moderation warning] This signature violated the rule against commercial exhortations.