STS (v3.0) - Knight Outposts/Centralization/Repositioning

[swami]

Crafty seems on par with Rybka for this test set.
It seems that Crafty has excellent knight understanding.

Or Bob has an ultra-superior hardware at his disposal. Let me run to see how Crafty does on my Core2Quad...

What's the command line to extend the hash size of Crafty to 256 mb in Arena? It's using the default 9 mb. I only have 32 bits though and little slower hardware relative to what Bob has, so the results may not appear close to 85's anyway.

[MattieShoes]

Moneypenny 10 seconds:

Code: Select all

Number of tests:                       100
Max score:                             1000
Engines score:                         602
Number with full score:                45
Found in: 5 sec = 15, 10 sec = 30,
Time used for tests (wrong answer=10s): 811s

Moneypenny 2 minutes:

Code: Select all

Number of tests:                       100
Max score:                             1000
Engines score:                         641
Number with full score:                54
Found in: 1 min = 15, 2 min = 39,
Time used for tests (wrong answer=120s): 9685s

[bob]

Crafty seems on par with Rybka for this test set.
It seems that Crafty has excellent knight understanding.

Or Bob has an ultra-superior hardware at his disposal. Let me run to see how Crafty does on my Core2Quad...

I ran on both the 8-core box I have been using for 3 years or so as well as on my core2 duo laptop. Not much difference. The 8-core is 2.33ghz, my laptop is 2.0ghz.

[bob]

Crafty seems on par with Rybka for this test set.
It seems that Crafty has excellent knight understanding.

Or Bob has an ultra-superior hardware at his disposal. Let me run to see how Crafty does on my Core2Quad...

What's the command line to extend the hash size of Crafty to 256 mb in Arena? It's using the default 9 mb. I only have 32 bits though and little slower hardware relative to what Bob has, so the results may not appear close to 85's anyway.

You can edit your crafty.rc file and add hash=384M or 192M (256M is not possible in Crafty).

[bob]

Chapter 3 Of the Strategic Test Suite: Knight Outposts, Centralization and Repositioning is now available for download.

* Consists of carefully selected 100 questionairres on Knight OutPosts/Centralization/Repositioning

* All of the questions in this test suite are selected by me and thoroughly verified by Dann Corbitt. Each question has been analysed by top engines for hours.

* All the answers carry similar traits:

• (1) A Knight move to take control of the center or posit itself in the center. (Centralization)
  (2) A Knight move to step back and reposition itself to lead the different (or better) path. (Repositioning)
  (3) A Knight move to take control of the opponent's weak squares (Outposts)

I initially selected about 190 problems and sent them to Dann, of which 100 has passed criteria. Dann made the final selection and done all that analysis with the help of the engines.

Recommended time: Anywhere between 10 seconds - 2 minutes for each test.

Download it here: STS 3.0
http://computerchessblogger.googlepages.com/sts

Feel free to report the results from your engine/ favourite engine, suggestions are welcome.

Special Thanks to Pedro Castro, Allard Siemelink, Zach Wegner for the testing and correcting mistakes throughout the process. And Thanks to those who reported bugs and tested out the V 1.0 of the test suite called "Undermining", and v2.0 called "Open Files and Diagonals".

And Extra Special Thanks to Ulysses for creating the beautiful logo!

Test Suite Released time: 3rd of June, 2009
Swaminathan and Dann Corbit.

I ran a quick test with the following details:

10 seconds per move.

I told Crafty that if it produced the same (correct) move for 5 consecutive iterations, to consider this one done and move on to the next position. That speeds things up quite a bit although I can run without it if it matters.

Here's the results:

Code: Select all

total positions searched..........         100
number right......................          96
number wrong......................           4
percentage right..................          96
percentage wrong..................           4
total nodes searched..............  1430919140
average search depth..............         8.1
nodes per second..................    15638460
total time........................        1:31

Crafty missed #13 (chose Nd5), #17 (liked Ncxe4), #57 (likes Qg6), #93 (likes Nd5)

Ran it again at 30 secs/position, but had to keep correct move for 6 iterations before stopping. Got 97 out of 100. Picked up #93...

Thanks for the test run, Bob. That's very impressive result, may I know what hardware was used?

dual socket 4-core intel, 2.33ghz, the same machine I have been using for about 3 years now...

I ran it on my 2.0ghz laptop and will post those numbers later when I get home..

[bob]

Here's the results from my core2-duo 2.0ghz laptop, 30 seconds per move, stopping when it has the correct move for six consecutive iterations, or else after 30 seconds have elapsed.

Code: Select all

test results summary:

total positions searched..........         100
number right......................          95
number wrong......................           5
percentage right..................          95
percentage wrong..................           5
total nodes searched..............  1544007086
average search depth..............         9.3
nodes per second..................     3847992
total time........................        6:41

[bob]

Since I have seen some odd results Here are the three sets of positions run on my 8-core box. Order is undermine, then file-diag, then knights. 30 secs/position, stop if correct move is chosen on 6 consecutive iterations:

Code: Select all

total positions searched..........         100
number right......................          85
number wrong......................          15
percentage right..................          85
percentage wrong..................          15
total nodes searched.............. 16161816467
average search depth..............        13.1
nodes per second..................    16536365
total time........................       16:17

total positions searched..........         100
number right......................          89
number wrong......................          11
percentage right..................          89
percentage wrong..................          11
total nodes searched.............. 17665398896
average search depth..............        13.2
nodes per second..................    17172546
total time........................       17:08

total positions searched..........         100
number right......................          96
number wrong......................           4
percentage right..................          96
percentage wrong..................           4
total nodes searched..............  3482397961
average search depth..............         9.1
nodes per second..................    16628774
total time........................        3:29

based on the search time needed, the last batch is by far the easiest...

[bob]

Crafty seems on par with Rybka for this test set.
It seems that Crafty has excellent knight understanding.

Or Bob has an ultra-superior hardware at his disposal. Let me run to see how Crafty does on my Core2Quad...

What's the command line to extend the hash size of Crafty to 256 mb in Arena? It's using the default 9 mb. I only have 32 bits though and little slower hardware relative to what Bob has, so the results may not appear close to 85's anyway.

You can edit your crafty.rc file and add hash=384M or 192M (256M is not possible in Crafty).

BTW after running on my laptop with just 2 cores at 2ghz, you ought to be able to beat the 95 I see on this machine, at least...

[Dann Corbit]

Since I have seen some odd results Here are the three sets of positions run on my 8-core box. Order is undermine, then file-diag, then knights. 30 secs/position, stop if correct move is chosen on 6 consecutive iterations:

Code: Select all

total positions searched..........         100
number right......................          85
number wrong......................          15
percentage right..................          85
percentage wrong..................          15
total nodes searched.............. 16161816467
average search depth..............        13.1
nodes per second..................    16536365
total time........................       16:17

total positions searched..........         100
number right......................          89
number wrong......................          11
percentage right..................          89
percentage wrong..................          11
total nodes searched.............. 17665398896
average search depth..............        13.2
nodes per second..................    17172546
total time........................       17:08

total positions searched..........         100
number right......................          96
number wrong......................           4
percentage right..................          96
percentage wrong..................           4
total nodes searched..............  3482397961
average search depth..............         9.1
nodes per second..................    16628774
total time........................        3:29

based on the search time needed, the last batch is by far the easiest...

I've seen other fairly strong programs have equal problems with this set that they have had with the others. I think crafty is just really good with knight moves for whatever reason.

[bob]

Since I have seen some odd results Here are the three sets of positions run on my 8-core box. Order is undermine, then file-diag, then knights. 30 secs/position, stop if correct move is chosen on 6 consecutive iterations:

Code: Select all

total positions searched..........         100
number right......................          85
number wrong......................          15
percentage right..................          85
percentage wrong..................          15
total nodes searched.............. 16161816467
average search depth..............        13.1
nodes per second..................    16536365
total time........................       16:17

total positions searched..........         100
number right......................          89
number wrong......................          11
percentage right..................          89
percentage wrong..................          11
total nodes searched.............. 17665398896
average search depth..............        13.2
nodes per second..................    17172546
total time........................       17:08

total positions searched..........         100
number right......................          96
number wrong......................           4
percentage right..................          96
percentage wrong..................           4
total nodes searched..............  3482397961
average search depth..............         9.1
nodes per second..................    16628774
total time........................        3:29

based on the search time needed, the last batch is by far the easiest...

I've seen other fairly strong programs have equal problems with this set that they have had with the others. I think crafty is just really good with knight moves for whatever reason.

I am not sure why. It's knight scoring is about as simple as it can get...