help validating perft numbers

Discussion of chess software programming and technical issues.

Moderator: Ras

jesper_nielsen

Re: help validating perft numbers

Post by jesper_nielsen »

jesper_nielsen wrote:
xcomponent wrote:

Code: Select all

Pawny 0.13
setboard 2K5/3P4/2k1q3/8/3R4/8/8/8 b - - 14 120
perft 8

-----------------------
 depth   nodes
-----------------------
     1   24
     2   360
     3   7554
     4   123671
     5   2611766
     6   45554896
     7   969124580
     8   17883914386
-----------------------
Confirmed ! :wink:
Brilliant! Thanks! :)

/jesper
I have matched your numbers for depth 7 and 8. Thanks all!

I won't tell you how long it took ... I am not green with envy at your times ... :( ... it is just that i have decided that speed is not that important ... yes ... that is the reason ... definitely ... :(

Kind regards,
Jesper
User avatar
Zach Wegner
Posts: 1922
Joined: Thu Mar 09, 2006 12:51 am
Location: Earth

Re: help validating perft numbers

Post by Zach Wegner »

hgm wrote:1.3GHz Celeron-M laptop:

Code: Select all

$ ./qperft 8 H23 "2K5/3P4/2k1q3/8/3R4/8/8/8 b - - 0 1"
Hash-table size = 7fffff, Starts at 4a0040
 - - - - - - - - - - - -
 - - - - - - - - - - - -
 - - . . K . . . . . - -
 - - . . . P . . . . - -
 - - . . k . q . . . - -
 - - . . . . . . . . - -
 - - . . . R . . . . - -
 - - . . . . . . . . - -
 - - . . . . . . . . - -
 - - . . . . . . . . - -
 - - - - - - - - - - - -
 - - - - - - - - - - - -

Quick Perft by H.G. Muller
Perft mode: Hash-table size = 128MB, bulk counting in horizon nodes

perft(1)=24 ( 0.000 sec)        probes=       0, entries=       0 hits=       0

perft(2)=360 ( 0.000 sec)       probes=      29, entries=      29 hits=       0

perft(3)=7554 ( 0.010 sec)      probes=     390, entries=     419 hits=       0

perft(4)=123671 ( 0.010 sec)    probes=    9479, entries=    4778 hits=    1866

perft(5)=2611766 ( 0.120 sec)   probes=   49338, entries=   35832 hits=   18162

perft(6)=45554896 ( 0.781 sec)  probes=  709196, entries=  196203 hits=  371924

perft(7)=969124580 ( 2.123 sec) probes= 2064426, entries=  887523 hits= 1284936

perft(8)=17883914386 (14.351 sec)       probes=18760156, entries= 2743564 hits=13414405
Hashing is a really big help in this position (obviously).

Results on a 2.4 GHz Q6600 here.

Code: Select all

(zct)1... perft 7
moves=969124580 time=0.679
(zct)1... perft 8
moves=17883914386 time=4.537
(zct)1... perft 9
moves=380865893753 time=12.411
(zct)1... perft 10
moves=7423954315810 time=1:11
...these were done in separate runs too so that the hashing from one perft wouldn't speed up the next.

For comparison, here's qperft on the same machine:

Code: Select all

perft(7)=969124580 ( 0.630 sec)

perft(8)=17883914386 ( 5.300 sec)

perft(9)=380865893753 (16.090 sec)

perft(10)=7423954315810 (140.730 sec)
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: help validating perft numbers

Post by bob »

Zach Wegner wrote:
hgm wrote:1.3GHz Celeron-M laptop:

Code: Select all

$ ./qperft 8 H23 "2K5/3P4/2k1q3/8/3R4/8/8/8 b - - 0 1"
Hash-table size = 7fffff, Starts at 4a0040
 - - - - - - - - - - - -
 - - - - - - - - - - - -
 - - . . K . . . . . - -
 - - . . . P . . . . - -
 - - . . k . q . . . - -
 - - . . . . . . . . - -
 - - . . . R . . . . - -
 - - . . . . . . . . - -
 - - . . . . . . . . - -
 - - . . . . . . . . - -
 - - - - - - - - - - - -
 - - - - - - - - - - - -

Quick Perft by H.G. Muller
Perft mode: Hash-table size = 128MB, bulk counting in horizon nodes

perft(1)=24 ( 0.000 sec)        probes=       0, entries=       0 hits=       0

perft(2)=360 ( 0.000 sec)       probes=      29, entries=      29 hits=       0

perft(3)=7554 ( 0.010 sec)      probes=     390, entries=     419 hits=       0

perft(4)=123671 ( 0.010 sec)    probes=    9479, entries=    4778 hits=    1866

perft(5)=2611766 ( 0.120 sec)   probes=   49338, entries=   35832 hits=   18162

perft(6)=45554896 ( 0.781 sec)  probes=  709196, entries=  196203 hits=  371924

perft(7)=969124580 ( 2.123 sec) probes= 2064426, entries=  887523 hits= 1284936

perft(8)=17883914386 (14.351 sec)       probes=18760156, entries= 2743564 hits=13414405
Hashing is a really big help in this position (obviously).

Results on a 2.4 GHz Q6600 here.

Code: Select all

(zct)1... perft 7
moves=969124580 time=0.679
(zct)1... perft 8
moves=17883914386 time=4.537
(zct)1... perft 9
moves=380865893753 time=12.411
(zct)1... perft 10
moves=7423954315810 time=1:11
...these were done in separate runs too so that the hashing from one perft wouldn't speed up the next.

For comparison, here's qperft on the same machine:

Code: Select all

perft(7)=969124580 ( 0.630 sec)

perft(8)=17883914386 ( 5.300 sec)

perft(9)=380865893753 (16.090 sec)

perft(10)=7423954315810 (140.730 sec)
While this is all well and good, let me remind you that _I_ originally came up with this metric. And it originally was intended to be a simple performance measurement for the efficiency of move generation, + make and unmake, and legal move verification. Hashing is invalid in that context. Peter McKenzie then published a position that was useful for testing the correctness of a move generator by using the perft node count.

Somewhere along the way, some started trying to find ways to speed it up, when that was never the goal other than you could use this to speed up make/unmake/movgen/legal-status-checker. But hashing? :) No point. This stuff is useless in a real game anyway.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: help validating perft numbers

Post by bob »

stegemma wrote:
bob wrote:...Here's my numbers:

Black(1): perft 6
total moves=45554896 time=3.50
Black(1): perft 7
total moves=969124580 time=69.68
Black(1): perft 8
total moves=17883914386 time=1311.43

This on a 2.0ghz laptop. I don't do the perft hashing trick, so I really traverse and make/unmake each move as well as do the move generations and check them for legality.
I'm curious me too. Even Freccia does a complete search with full make/unmake and a full "slow" test for check at any move (for check it tests all the possible squares from the King, as to generates queen's moves plus knight moves). My time is about 1100 seconds. About half the time can be reached with a better check test algorithm than mine. Maybe with transposition tables and avoiding make/unmake move at the last node can give even more better results, i think. For me, having a perft better than Crafty (on the same machine) is more than expected and i don't optimize further, until the other algorithm will be completed.
First question is, what is your board representation? Bitboards have advantages and disadvantages. One significant advantage is in the evaluation where you can ask multiple questions by doing boolean operations on "sets" of information in one instruction. Make/Unmake is not particularly fast compared to a 0x88-like implementation because of redundant work being done (occupied squares, etc). The first place bitboards shows its versatility is when you choose to generate just captures, as here it is simply faster than 0x88. But perft doesn't do a capture-only search so it plays no part in perft.

If I were trying to write the fastest-possible perft, I would use something else. I use perft to check things after making a significant change to the move generator, or make/unmake, or the check detector.
Dann Corbit
Posts: 12792
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: help validating perft numbers

Post by Dann Corbit »

bob wrote:
Zach Wegner wrote:
hgm wrote:1.3GHz Celeron-M laptop:

Code: Select all

$ ./qperft 8 H23 "2K5/3P4/2k1q3/8/3R4/8/8/8 b - - 0 1"
Hash-table size = 7fffff, Starts at 4a0040
 - - - - - - - - - - - -
 - - - - - - - - - - - -
 - - . . K . . . . . - -
 - - . . . P . . . . - -
 - - . . k . q . . . - -
 - - . . . . . . . . - -
 - - . . . R . . . . - -
 - - . . . . . . . . - -
 - - . . . . . . . . - -
 - - . . . . . . . . - -
 - - - - - - - - - - - -
 - - - - - - - - - - - -

Quick Perft by H.G. Muller
Perft mode: Hash-table size = 128MB, bulk counting in horizon nodes

perft(1)=24 ( 0.000 sec)        probes=       0, entries=       0 hits=       0

perft(2)=360 ( 0.000 sec)       probes=      29, entries=      29 hits=       0

perft(3)=7554 ( 0.010 sec)      probes=     390, entries=     419 hits=       0

perft(4)=123671 ( 0.010 sec)    probes=    9479, entries=    4778 hits=    1866

perft(5)=2611766 ( 0.120 sec)   probes=   49338, entries=   35832 hits=   18162

perft(6)=45554896 ( 0.781 sec)  probes=  709196, entries=  196203 hits=  371924

perft(7)=969124580 ( 2.123 sec) probes= 2064426, entries=  887523 hits= 1284936

perft(8)=17883914386 (14.351 sec)       probes=18760156, entries= 2743564 hits=13414405
Hashing is a really big help in this position (obviously).

Results on a 2.4 GHz Q6600 here.

Code: Select all

(zct)1... perft 7
moves=969124580 time=0.679
(zct)1... perft 8
moves=17883914386 time=4.537
(zct)1... perft 9
moves=380865893753 time=12.411
(zct)1... perft 10
moves=7423954315810 time=1:11
...these were done in separate runs too so that the hashing from one perft wouldn't speed up the next.

For comparison, here's qperft on the same machine:

Code: Select all

perft(7)=969124580 ( 0.630 sec)

perft(8)=17883914386 ( 5.300 sec)

perft(9)=380865893753 (16.090 sec)

perft(10)=7423954315810 (140.730 sec)
While this is all well and good, let me remind you that _I_ originally came up with this metric. And it originally was intended to be a simple performance measurement for the efficiency of move generation, + make and unmake, and legal move verification. Hashing is invalid in that context. Peter McKenzie then published a position that was useful for testing the correctness of a move generator by using the perft node count.

Somewhere along the way, some started trying to find ways to speed it up, when that was never the goal other than you could use this to speed up make/unmake/movgen/legal-status-checker. But hashing? :) No point. This stuff is useless in a real game anyway.
I think it could be valid in some contexts. Consider mate solving mode, where all we care about is checkmate and we are using a proof search.

If we add a simple eval that only determines, mate/draw/-mate it seems that an ultra fast hashing move generator + ultra-simple eval could be genuinely useful.

It could also have utility in game play with a cluster scenario, as one CPU might be dedicated to mate searches along the pv.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: help validating perft numbers

Post by bob »

Dann Corbit wrote:
bob wrote:
Zach Wegner wrote:
hgm wrote:1.3GHz Celeron-M laptop:

Code: Select all

$ ./qperft 8 H23 "2K5/3P4/2k1q3/8/3R4/8/8/8 b - - 0 1"
Hash-table size = 7fffff, Starts at 4a0040
 - - - - - - - - - - - -
 - - - - - - - - - - - -
 - - . . K . . . . . - -
 - - . . . P . . . . - -
 - - . . k . q . . . - -
 - - . . . . . . . . - -
 - - . . . R . . . . - -
 - - . . . . . . . . - -
 - - . . . . . . . . - -
 - - . . . . . . . . - -
 - - - - - - - - - - - -
 - - - - - - - - - - - -

Quick Perft by H.G. Muller
Perft mode: Hash-table size = 128MB, bulk counting in horizon nodes

perft(1)=24 ( 0.000 sec)        probes=       0, entries=       0 hits=       0

perft(2)=360 ( 0.000 sec)       probes=      29, entries=      29 hits=       0

perft(3)=7554 ( 0.010 sec)      probes=     390, entries=     419 hits=       0

perft(4)=123671 ( 0.010 sec)    probes=    9479, entries=    4778 hits=    1866

perft(5)=2611766 ( 0.120 sec)   probes=   49338, entries=   35832 hits=   18162

perft(6)=45554896 ( 0.781 sec)  probes=  709196, entries=  196203 hits=  371924

perft(7)=969124580 ( 2.123 sec) probes= 2064426, entries=  887523 hits= 1284936

perft(8)=17883914386 (14.351 sec)       probes=18760156, entries= 2743564 hits=13414405
Hashing is a really big help in this position (obviously).

Results on a 2.4 GHz Q6600 here.

Code: Select all

(zct)1... perft 7
moves=969124580 time=0.679
(zct)1... perft 8
moves=17883914386 time=4.537
(zct)1... perft 9
moves=380865893753 time=12.411
(zct)1... perft 10
moves=7423954315810 time=1:11
...these were done in separate runs too so that the hashing from one perft wouldn't speed up the next.

For comparison, here's qperft on the same machine:

Code: Select all

perft(7)=969124580 ( 0.630 sec)

perft(8)=17883914386 ( 5.300 sec)

perft(9)=380865893753 (16.090 sec)

perft(10)=7423954315810 (140.730 sec)
While this is all well and good, let me remind you that _I_ originally came up with this metric. And it originally was intended to be a simple performance measurement for the efficiency of move generation, + make and unmake, and legal move verification. Hashing is invalid in that context. Peter McKenzie then published a position that was useful for testing the correctness of a move generator by using the perft node count.

Somewhere along the way, some started trying to find ways to speed it up, when that was never the goal other than you could use this to speed up make/unmake/movgen/legal-status-checker. But hashing? :) No point. This stuff is useless in a real game anyway.
I think it could be valid in some contexts. Consider mate solving mode, where all we care about is checkmate and we are using a proof search.

If we add a simple eval that only determines, mate/draw/-mate it seems that an ultra fast hashing move generator + ultra-simple eval could be genuinely useful.

It could also have utility in game play with a cluster scenario, as one CPU might be dedicated to mate searches along the pv.
There is more too it. A real search goes deeper. It actually executes some complex code dealing with repetitions and such, which perft does not do. Bottom line is that perft executes less than 10% of the total code used to do a real search, which doesn't really make perft times worth comparing. NPS is much more interesting in a real search. With perft, the whole thing can fit in L1 easily. In a real program, this doesn't happen.
User avatar
xsadar
Posts: 147
Joined: Wed Jun 06, 2007 10:01 am
Location: United States
Full name: Mike Leany

Re: help validating perft numbers

Post by xsadar »

bob wrote: While this is all well and good, let me remind you that _I_ originally came up with this metric. And it originally was intended to be a simple performance measurement for the efficiency of move generation, + make and unmake, and legal move verification. Hashing is invalid in that context. Peter McKenzie then published a position that was useful for testing the correctness of a move generator by using the perft node count.

Somewhere along the way, some started trying to find ways to speed it up, when that was never the goal other than you could use this to speed up make/unmake/movgen/legal-status-checker. But hashing? :) No point. This stuff is useless in a real game anyway.
In my opinion, using a perft-like function to test the correctness of the move generator is far more useful and important than using it to test the performance (even though that was it's original purpose). So with that in mind, hashing can serve a very useful purpose. Why would we want to waste time counting positions that we've already counted if all we care about is comparing counts so we can either debug, or verify the correctness of the move generator? Hashing can alleviate that problem.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: help validating perft numbers

Post by bob »

xsadar wrote:
bob wrote: While this is all well and good, let me remind you that _I_ originally came up with this metric. And it originally was intended to be a simple performance measurement for the efficiency of move generation, + make and unmake, and legal move verification. Hashing is invalid in that context. Peter McKenzie then published a position that was useful for testing the correctness of a move generator by using the perft node count.

Somewhere along the way, some started trying to find ways to speed it up, when that was never the goal other than you could use this to speed up make/unmake/movgen/legal-status-checker. But hashing? :) No point. This stuff is useless in a real game anyway.
In my opinion, using a perft-like function to test the correctness of the move generator is far more useful and important than using it to test the performance (even though that was it's original purpose). So with that in mind, hashing can serve a very useful purpose. Why would we want to waste time counting positions that we've already counted if all we care about is comparing counts so we can either debug, or verify the correctness of the move generator? Hashing can alleviate that problem.
Once your move generator is done, it is done. What's the reasoning behind spending time adding code (say perft hashing) that won't make the engine play any better at all. I haven't run a perft in years except for when someone asks for numbers for a specific engine so they have good data to compare against.
User avatar
hgm
Posts: 28391
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: help validating perft numbers

Post by hgm »

But qperft is not an engine at all. Who wants an engine? It is evaluation that is completely useless. It does not help you at all to count move paths...

As usual you ar arguing from the tunnel vision of your own limited interests.
User avatar
stegemma
Posts: 859
Joined: Mon Aug 10, 2009 10:05 pm
Location: Italy
Full name: Stefano Gemma

Re: help validating perft numbers

Post by stegemma »

bob wrote:...
First question is, what is your board representation? Bitboards have advantages and disadvantages. One significant advantage is in the evaluation where you can ask multiple questions by doing boolean operations on "sets" of information in one instruction. Make/Unmake is not particularly fast compared to a 0x88-like implementation because of redundant work being done (occupied squares, etc). The first place bitboards shows its versatility is when you choose to generate just captures, as here it is simply faster than 0x88. But perft doesn't do a capture-only search so it plays no part in perft.

If I were trying to write the fastest-possible perft, I would use something else. I use perft to check things after making a significant change to the move generator, or make/unmake, or the check detector.
I use a 16x10 board. , that means:

Code: Select all


I=Illegal
V=Vuota (empty)

I,I,I,I,I,I,I,I,I,I
I,I,I,I,I,I,I,I,I,I
I,I,I,I,I,I,I,I,I,I
I,I,I,I,I,I,I,I,I,I
V,V,V,V,V,V,V,V,I,I
V,V,V,V,V,V,V,V,I,I
V,V,V,V,V,V,V,V,I,I
V,V,V,V,V,V,V,V,I,I
V,V,V,V,V,V,V,V,I,I
V,V,V,V,V,V,V,V,I,I
V,V,V,V,V,V,V,V,I,I
V,V,V,V,V,V,V,V,I,I
I,I,I,I,I,I,I,I,I,I
I,I,I,I,I,I,I,I,I,I
The first 2 illegal squares lines are more than needed, one extra illegal square is enough to handle knight on a8 illegal moves (on the left top square). I don't use board indexes but directly square pointers. Its easiest to handle, in assembly, from my point of view. Despite from that, it is a common rapresentation, i think.

Maybe transposition tables used in standard search and applyed to perft can help to debug the transposition tables itself or to collect statistics? In fact, as you said, there are no usefull raison to speed up perft but it becomes interesting if the speed up one is trying can then be applyed to normal search. Using some standard search speed-up in perft can give an information about that speed-up correctness.

Coming back to check test, i test the squares to know if there are an enemy piece of the required types. I test many piece types for any square tested, depending if they are ne suqare near or more to the king. Supposing to have to test for check to a white king in e4, i test:

d5,f5 for K/Q/B/P
e5 for K/Q/R
d3,f3 for K/Q/B
e3 for K/Q

then others "e" column squares for Q/R and "5" line for Q/R and obviously the 2 diagonals for Q/B.

When i find a friend piece or an enemy piece not of desired type i stop testing on that line (column or diagonal).

At end, i test for 8 knights squares.

The test costs "only" 3 assembly instructions per square (the test is in something like an unrolled loop, in assembly - in fact i use macros to write readable code). It is very expensive because it requires testing for 16/17 squares for a just castled king with all pawns unmoved. It means about 50 instructions. If the king is in the middle of the board, it takes more inst. Generating a single moves takes me about 7/8 inst. so the cost of testing for check is about as to generate almost 7 more moves per any move played!!! Obviously making the moves takes more than generate it... so the effective cost, compared with move generation, is less important but still higher than other approach, i think.

In previous programs (Drago/Raffaela) i tested for check only at the next ply (that means not to check at all) but with Freccia i want to try to leave the check as soon as i have played the move (or a little before, if i want further optimize).