TalkChess.com
Hosted by Your Move Chess & Games

 Comparative nodes per second Goto page 1, 2, 3 ... 12, 13, 14  Next
Author Message
Larry Kaufman

Joined: 10 Jan 2010
Posts: 1228
Location: Maryland USA

 Posted: Tue Apr 10, 2012 7:49 pm    Post subject: Comparative nodes per second Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer. So, two questions: 1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish? 2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?
Vincent Diepeveen

Joined: 09 Mar 2006
Posts: 1738
Location: The Netherlands

Posted: Tue Apr 10, 2012 8:16 pm    Post subject: Re: Comparative nodes per second

 lkaufman wrote: Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer. So, two questions: 1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish? 2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?

In Diep i'm not counting illegal positions after a check as it has incremental attacktables. So i kick that illegal move out already. Most engines aren't doing this, some are. Also i'm prematurely detecting repetitions. If i drop that condition the NPS already increases factor 2 at many occasions of Diep.

So Diep's nps is in majority of positions about a factor 2 lower than when i would count it in the same manner like others, just because it detects things very early - of course i can do that as i already blow a lot of system time to datastructure there - so early detection in the speedy programs would be a very dumb idea to think about. Yet a seemingly 'small' counting decision there can result easily in factor 2 nps difference.

Then further the engines you quote are doing different types of forward pruning. This really can hammer the nps down by factors. Stockfish is doing an agressive form of futility for example - i'm not sure Ivanhoe is doing that (didn't check ivanhoe). But this can really matter factors in nps. When i say factors i don't mean 20% or 30%. I really mean factor 2 or factor 3.

You just totally cannot compare nps-es in an easy manner.

Junior on other hand is *adding* the futile pruned moves to its nps. So that's why its nps seems high.

However if you really would neglect the forward pruning they do last few plies, most engines are not much above 1000 cycles a node. So that would mean for a StockFish it has the potential to reach at a 3.6Ghz i7 with sixcores somehwere around a 6 * 3.6 mln = roughly 20 mln nps.

Vincent
Marco Costalba

Joined: 14 Jun 2008
Posts: 2090

Posted: Tue Apr 10, 2012 8:41 pm    Post subject: Re: Comparative nodes per second

 lkaufman wrote: I don't believe that such details would amount to more than a few percent in the given instance.

It's time to became a believer

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.

P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.
Larry Kaufman

Joined: 10 Jan 2010
Posts: 1228
Location: Maryland USA

Posted: Tue Apr 10, 2012 9:02 pm    Post subject: Re: Comparative nodes per second

mcostalba wrote:
 lkaufman wrote: I don't believe that such details would amount to more than a few percent in the given instance.

It's time to became a believer

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.

P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.

We count the node only when we make the move. If we can prune a move without making it, we don't count the node that would result from making the move. Are you saying that Ivanhoe counts nodes that would result from pruned moves that are never made? If so that would indeed inflate the node count. When Stockfish prunes moves by movecount or by futility are the resultant nodes counted or not?

I know that SF is a very fast program. That's why I want to clarify whether this 5 to 3 speed ratio is entirely due to counting issues or only partly so.
Thomas Petzke

Joined: 03 Mar 2011
Posts: 263
Location: Germany

 Posted: Tue Apr 10, 2012 9:30 pm    Post subject: Re: Comparative nodes per second If you count in do_move then you count moves not nodes, of course they are somehow related because every move results in a new node but you don't count terminal nodes (where you only call eval, get a standpat or a hash hit with a cutoff). This should make a big difference. Thomas...
Robert Hyatt

Joined: 27 Feb 2006
Posts: 15816
Location: Birmingham, AL

Posted: Tue Apr 10, 2012 9:31 pm    Post subject: Re: Comparative nodes per second

 lkaufman wrote: Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer. So, two questions: 1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish? 2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?

Have you compared to Crafty? I know exactly how I count nodes, which would give a starting point that doesn't require a lot of effort to figure out...
Larry Kaufman

Joined: 10 Jan 2010
Posts: 1228
Location: Maryland USA

Posted: Tue Apr 10, 2012 9:35 pm    Post subject: Re: Comparative nodes per second

bob wrote:
 lkaufman wrote: Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer. So, two questions: 1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish? 2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?

Have you compared to Crafty? I know exactly how I count nodes, which would give a starting point that doesn't require a lot of effort to figure out...

Since Crafty doesn't do the massive move-count based pruning that is used in all the top engines, I don't believe the node-counting in Crafty would be very relevant to node-counting in these engines, where the counting or not of these pruned moves is so critical.
Richard Vida

Joined: 16 Apr 2009
Posts: 467
Location: Slovakia, EU

Posted: Tue Apr 10, 2012 10:15 pm    Post subject: Re: Comparative nodes per second

mcostalba wrote:
 lkaufman wrote: I don't believe that such details would amount to more than a few percent in the given instance.

It's time to became a believer

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.

P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.

Ivanhoe updates nodes in do_move(). But because the legality check is done only after do_move(), there are some illegal moves included in Ivanhoe's node count.
Larry Kaufman

Joined: 10 Jan 2010
Posts: 1228
Location: Maryland USA

Posted: Tue Apr 10, 2012 10:18 pm    Post subject: Re: Comparative nodes per second

rvida wrote:
mcostalba wrote:
 lkaufman wrote: I don't believe that such details would amount to more than a few percent in the given instance.

It's time to became a believer

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.

P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.

Ivanhoe updates nodes in do_move(). But because the legality check is done only after do_move(), there are some illegal moves included in Ivanhoe's node count.

I think this issue accounts for just a few percent. Am I mistaken?
Sam Hamilton

Joined: 23 Jun 2006
Posts: 1095

 Posted: Tue Apr 10, 2012 10:54 pm    Post subject: Re: Comparative nodes per second I suspect much more time is spent in Stockfish eval than Ivanhoe. If I remember right, stockfish does not do lazy eval for example. It may even call eval every node (that eval is not cached) as well. The Ivan's try very hard to go fast by minimizing time spent in eval I think. Almost everyone in this discussion have looked at both programs more than I have, so feel free to correct any mistaken impressions I might have. -Sam
 Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First
 All times are GMTGoto page 1, 2, 3 ... 12, 13, 14  Next Page 1 of 14

 Jump to: Select a forum Computer Chess Club Forums----------------Computer Chess Club: General TopicsComputer Chess Club: Tournaments and MatchesComputer Chess Club: Programming and Technical DiscussionsComputer Chess Club: Engine Origins Other Forums----------------Chess Thinkers ForumForum Help and Suggestions
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum