So much for the importance of tactical strength

[Dann Corbit]

I gave a bunch of strong programs WACNEW.EPD (WAC + Corrections) and let several strong programs run at 6 seconds per position. The worst one for solution count was... Rybka!

I used to think that very strong, rapid tactical ability was a necessary but not sufficient condition for a very strong chess engine. Now I wonder if it is not nearly so important as I might have imagined.

Of course, given tournament time controls, the strong engines will do very well at all of these problems, because 295 of them or so are pretty easy.

Code: Select all

Analyzing engine: Alaric707
230) .. Rb7-b4          Rb7-h7   
243) Qf2-e2             h2-h3    
298 of 300 matching moves

Analyzing engine: Aristarch 4.50
2) .. Rb3xb2          Rb3-b7   
100) Bd2-e3, b5-b6, b5-b6  Ka4-b3   
131) Re1-e8             Bd4-f6   
141) Qc1xf4             Kg2-f1   
163) .. Qf3-g2          c6xd5    
196) .. Nc6-b4          Nf6-g4   
214) Ne4-g5             Qe2-e3   
222) Bh4-f6             b2xa3    
230) .. Rb7-b4          Rb7-h7   
243) Qf2-e2             b4-b5    
256) Nd4-f5             g2-g3    
270) Qd4-g4             Re2xe8   
274) Rg6-b6, Rg6-g5     Rg6-d6   
293) Nf3-g5             e2-e3    
286 of 300 matching moves

Analyzing engine: ChessTiger2007UCI
163) .. Qf3-g2          c6-c5    
299 of 300 matching moves

Analyzing engine: Delfi
81) .. Be7-d6          Bb7-e4   
163) .. Qf3-g2          c6xd5    
230) .. Rb7-b4          Rb7-h7   
297 of 300 matching moves

Analyzing engine: EngineShredder9UCI
108) Qc5-e5             Qc5-d6   
163) .. Qf3-g2          c6xd5    
298 of 300 matching moves

Analyzing engine: Fruit-2-3-1
2) .. Rb3xb2          Rb3-b8   
100) Bd2-e3, b5-b6, b5-b6  Ka4-b3   
163) .. Qf3-g2          c6-c5    
183) Bg5-f6             f2-f4    
229) .. Rh5xh4, b5-b4   Rh5-h7   
230) .. Rb7-b4          Rb7-h7   
294 of 300 matching moves

Analyzing engine: Gandalf60
100) Bd2-e3, b5-b6, b5-b6  Ka4-b3   
230) .. Rb7-b4          Rb7-h7   
298 of 300 matching moves

Analyzing engine: Hiarcs11.2SPUCI
163) .. Qf3-g2          c6-c5    
214) Ne4-g5             Qe2-e3   
230) .. Rb7-b4          Rb7-h7   
297 of 300 matching moves

Analyzing engine: Ktulu_75
163) .. Qf3-g2          c6xd5    
230) .. Rb7-b4          Rb7-h7   
262) .. Rg6-h6          a7-a6    
293) Nf3-g5             e2-e3    
296 of 300 matching moves

Analyzing engine: List512
2) .. Rb3xb2          Rb3-b7   
100) Bd2-e3, b5-b6, b5-b6  Ka4-b3   
230) .. Rb7-b4          Rb7-h7   
293) Nf3-g5             e2-e3    
296 of 300 matching moves

Analyzing engine: Movei UCI
92) .. Be6xg4          Ra8-c8   
100) Bd2-e3, b5-b6, b5-b6  Ka4-b3   
230) .. Rb7-b4          Rb7-h7   
297 of 300 matching moves

Analyzing engine: Prodeo 1.6
2) .. Rb3xb2          Rb3-b7   
100) Bd2-e3, b5-b6, b5-b6  Ka4-b3   
163) .. Qf3-g2          c6xd5    
230) .. Rb7-b4          a5-a4    
243) Qf2-e2             Qf2-f1   
295 of 300 matching moves

Analyzing engine: Rybka v1.2.w32
2) .. Rb3xb2          Rb3-b8   
43) Ba3-e7, Qd5xa8     Ba3-c3   
56) .. Bc5xf2          g7-f8    
71) Nb5xa7             Qh3-f3   
89) .. g7-g5           Kh3-g2   
139) Ne4-f6             g7-g5    
161) Qh4xd8             ...      
163) .. Qf3-g2          c6-c5    
174) .. Nf4xg2          Rf8-d8   
175) Nf4-h5             Nf4-g2   
216) Ne5xf7, a2-a4      Bd3xh7   
230) .. Rb7-b4          Rb7-h7   
245) Qd3xg6             Qe3xc5   
253) Qe3-e8             Rg8-g7   
261) .. Nf3-d4          c7-d8    
264) .. Qe7-e5, Ra8-b8  f7xg8    
268) Re2-e8             Rc8xc7   
288) Nh5-f6             Bf7-c4   
293) Nf3-g5             Qh6-e3   
281 of 300 matching moves

Analyzing engine: Spike1.2
80) Rd1-a1             Kg1xh2   
100) Bd2-e3, b5-b6, b5-b6  Ka4-b3   
105) .. Bd6xg3, Qh5xh4  Qe2xh5   
163) .. Qf3-g2          c6xd5    
230) .. Rb7-b4          Rb7-h7   
271) Re6-d6             a2-a4    
294 of 300 matching moves

Analyzing engine: Strelka_1.8_UCI
86) .. Nf6-g4          a7-a6    
229) .. Rh5xh4, b5-b4   Rh5-h8   
230) .. Rb7-b4          Rb7-h7   
297 of 300 matching moves

Analyzing engine: Glaurung_win_32
140) Be5-c7, Rc3-c7     Ne4-f6   
163) .. Qf3-g2          c6-c5    
230) .. Rb7-b4          Rb7-h7   
297 of 300 matching moves

[YL84]

Good remark Dann !
As you know when you tune an engine for tactical test suites it performs
worse in real games. So that it could be great to have some "real game test suites" to test our engine
Yves

[Uri Blass]

I gave a bunch of strong programs WACNEW.EPD (WAC + Corrections) and let several strong programs run at 6 seconds per position. The worst one for solution count was... Rybka!

I used to think that very strong, rapid tactical ability was a necessary but not sufficient condition for a very strong chess engine. Now I wonder if it is not nearly so important as I might have imagined.

I think that one reason is that the wac suite is about sacrifices and rybka has big reductions for sacrifices.

Another reason is that you used the version of rybka that is the weakest for test suites and I expect that you get better result if you use the free rybka or you use Rybka2.3.2a

Tactics is not only about sacrifices and I guess if you use the arasan test suite rybka is going to perform relatively better(not sure about the latest arasan test suite that may be more about sacrifices and I suggest using the old arasan test suite).

Uri

[Piotr Cichy]

Maybe WAC is just too easy for typical chessengines nowadays?
NanoSzachy 3.0 is rather weak engine both in tactics and knowledge, but performs very well on this test: only 7 positions not solved.

[Mike S.]

WAC is too easy. But I agree to your opinion, at least partially. I think though, that a certain, good "tactical base speed", or fundament is required for any engine in the top-20, at least. Some may have that tactical strength but lack other things, so it doesn't mean they have to be top-20 in total.

In my tactical test, which is smaller but I think, in average much more difficult than WAC, Rybka 232a solved 17/24 positions (350 points) which indicates a tactical speed very similar to Fritz 7, Shredder 9 but also Aristarch 4.50. Hiarcs, King or Fruit solved 20 or 21. Note that Shredder 9, which was the strongest engine before Rybka emerged, also is not among the fastest combinators according to the Quicktest (unlike WAC).

Since most of the positions are "bm" in the active sense, Rybka WinFinder 2.2 achieved the best QT. result ever, solving 21 (458) the fastest.

http://members.aon.at/computerschach/quick/quicke.htm

Nevertheless, while being tactically even faster than the required base speed seems not to increase the overall playing strength in the top range, it is of course useful for analysing tactical positions.

[pedrox]

DanaSah 3.13 --> 297 of 300

Failure: 163, 230, 293

The positions 163 and 293 are mates, DanaSah need approximately 1 minute and 20 seconds.

Around position 230 DanaSah prefers Rh7 or a4 like most engines, I thought it was a position unresolved as I had read sometimes, but I am surprised that Shredder and ChessTiger it resolved.

Perhaps Rybka failure in the test tactical because it extends little the checks, It does not see many mates but goes deeper.

Pedro

[pedrox]

Dann, you can post analyzes of the position 230 for Shredder and ChessTiger?

Thx,

Pedro

[Uri Blass]

DanaSah 3.13 --> 297 of 300

Failure: 163, 230, 293

The positions 163 and 293 are mates, DanaSah need approximately 1 minute and 20 seconds.

Around position 230 DanaSah prefers Rh7 or a4 like most engines, I thought it was a position unresolved as I had read sometimes, but I am surprised that Shredder and ChessTiger it resolved.

Perhaps Rybka failure in the test tactical because it extends little the checks, It does not see many mates but goes deeper.

Pedro

I check most of the positions that rybka failed based on Dann Corbit's post and the main reason for rybka's bad result is probably the fact that rybka1.2 does not support perfectly analysis mode and is simply confused when it gets new positions so it analyze something different.

Dann Corbit gave the following analysis


Analyzing engine: Rybka v1.2.w32
2) .. Rb3xb2 Rb3-b8
43) Ba3-e7, Qd5xa8 Ba3-c3
56) .. Bc5xf2 g7-f8
71) Nb5xa7 Qh3-f3
89) .. g7-g5 Kh3-g2
139) Ne4-f6 g7-g5
161) Qh4xd8 ...
163) .. Qf3-g2 c6-c5
174) .. Nf4xg2 Rf8-d8
175) Nf4-h5 Nf4-g2
216) Ne5xf7, a2-a4 Bd3xh7
230) .. Rb7-b4 Rb7-h7
245) Qd3xg6 Qe3xc5
253) Qe3-e8 Rg8-g7
261) .. Nf3-d4 c7-d8
264) .. Qe7-e5, Ra8-b8 f7xg8
268) Re2-e8 Rc8xc7
288) Nh5-f6 Bf7-c4
293) Nf3-g5 Qh6-e3
281 of 300 matching moves

In 43 Ba3-c3 is simply illegal move
My rybkas have no problem to find the correct move
In 56 g7-f8 is simply illegal move
In 89 Kh3-g2 is illegal move(it is black to move and Kh3 is white)
In 139 g7-g5 is illegal move(it is white to move)
In 161 rybka gives no output
In 174 my rybka has no problem with the solution
In 175 we again have illegal move
In 216 rybka never considers Bxh7+
In 245 the move is illegal
In


My rybka1.2 could see legal moves easily during analysis mode so I guess that rybka has problems to understand the position that she got.

In 2 and 71 and 163 rybka really probably needs slightly more time and I did not check the rest of the failures in the time of this post.

Note also that the new rybka is faster in 71

Uri

[Steve Maughan]

Dann,

Interesting - IMO the Arasan suite is the best for testing game play tactics. It would be interesting to repeat the experiment with this suite.

Steve

[Jouni]

Sorry Dann, but ECM suite is totally useless today. Good engine like Rybka
2.3.2a solves practically all positions instantly!! I don't remember, what kind of version 1.2 was, but in my outdated PIV 2.3.2a solves 297/300 in 6 seconds. Not solved were 2 (19s) 163 (25s) and 230 (??). What use You
have of this (Fritz log):

WAC.003 0:00:00
WAC.004 0:00:00
WAC.005 0:00:00
WAC.006 0:00:00
WAC.007 0:00:00
WAC.008 0:00:01
WAC.009 0:00:00
WAC.010 0:00:01
WAC.011 0:00:01
WAC.012 0:00:00
WAC.013 0:00:00
WAC.014 0:00:01
WAC.015 0:00:00
WAC.016 0:00:00
WAC.017 0:00:00
WAC.018 0:00:00
WAC.019 0:00:01
WAC.020 0:00:01
WAC.021 0:00:01
WAC.022 0:00:00
WAC.023 0:00:01
WAC.024 0:00:00
WAC.025 0:00:01
WAC.026 0:00:00
WAC.027 0:00:00
WAC.028 0:00:00
WAC.029 0:00:01
WAC.030 0:00:01
WAC.031 0:00:00
WAC.032 0:00:01
WAC.033 0:00:01
WAC.034 0:00:00
WAC.035 0:00:00
WAC.036 0:00:00
WAC.037 0:00:00
WAC.038 0:00:00
WAC.039 0:00:00
WAC.040 0:00:00
WAC.041 0:00:00
WAC.042 0:00:00
WAC.043 0:00:01
WAC.044 0:00:00
WAC.045 0:00:00
WAC.046 0:00:00
WAC.047 0:00:01
WAC.048 0:00:01
WAC.049 0:00:01
WAC.050 0:00:00
WAC.051 0:00:01
WAC.052 0:00:00
WAC.053 0:00:00
WAC.054 0:00:00
WAC.055 0:00:01
WAC.056 0:00:01
WAC.057 0:00:00
WAC.058 0:00:01
WAC.059 0:00:00
WAC.060 0:00:01
WAC.061 0:00:00
WAC.062 0:00:00
WAC.063 0:00:01
WAC.064 0:00:00
WAC.065 0:00:01
WAC.066 0:00:01
WAC.067 0:00:00
WAC.068 0:00:01
WAC.069 0:00:00
WAC.070 0:00:01
WAC.071 0:00:01
WAC.072 0:00:01
WAC.073 0:00:01
WAC.074 0:00:00
WAC.075 0:00:00
WAC.076 0:00:01
WAC.077 0:00:01
WAC.078 0:00:00


Jouni