I downloaded Houdini...

[LaurenceChen]

But Houdini is designed to play Chess, and this is a legal Chess position...

Of course you could argue that this position is not likely to occur in a real game. (But of course, apart from positions just a few moves from the opening, no single position really is.)

Can you show me how would you get the position from a real game?

[michiguel]

But Houdini is designed to play Chess, and this is a legal Chess position...

Of course you could argue that this position is not likely to occur in a real game. (But of course, apart from positions just a few moves from the opening, no single position really is.)

Can you show me how would you get the position from a real game?

It could easily be created by underpromotion.

Miguel

[michiguel]

But Houdini is designed to play Chess, and this is a legal Chess position...

Of course you could argue that this position is not likely to occur in a real game. (But of course, apart from positions just a few moves from the opening, no single position really is.)

You are deliberately missing the point.

No, he is not. he is trying (unsuccessfully) to explain it. He is demonstrating how an AI entity which has been supposedly designed to play legal chess, fails miserably in one set of positions whereas it is extremely successful in others. Houdini, as almost all engines, is over-fitted. It is very nice to play chess, but it has a big problem with respect to AI.

Miguel

[K I Hyams]

But Houdini is designed to play Chess, and this is a legal Chess position...

Of course you could argue that this position is not likely to occur in a real game. (But of course, apart from positions just a few moves from the opening, no single position really is.)

You are deliberately missing the point.

No, he is not. he is trying (unsuccessfully) to explain it. He is demonstrating how an AI entity which has been supposedly designed to play legal chess, fails miserably in one set of positions whereas it is extremely successful in others. Houdini, as almost all engines, is over-fitted. It is very nice to play chess, but it has a big problem with respect to AI.

Miguel

An engine which can evaluate one type of position well may evaluate another type of position badly. This implies that as a series of positions are addressed that gradually stray further away from the “comfort zone” of a particular engine, the evaluation of that engine for those positions can be expected to become less reliable.

That implies to me that if you put Houdini, or any other engine, onto a position that is sufficiently alien then the evaluation will become virtually useless. I was assuming that, as the positions that HGM was showing Houdini were totally alien to a normal game of chess, the Houdini evaluation would not cope with them particularly well.

I suspect that I am not the only layman who might make those assumptions. If you would explain to me where they break down, I would be grateful.

[michiguel]

But Houdini is designed to play Chess, and this is a legal Chess position...

Of course you could argue that this position is not likely to occur in a real game. (But of course, apart from positions just a few moves from the opening, no single position really is.)

You are deliberately missing the point.

No, he is not. he is trying (unsuccessfully) to explain it. He is demonstrating how an AI entity which has been supposedly designed to play legal chess, fails miserably in one set of positions whereas it is extremely successful in others. Houdini, as almost all engines, is over-fitted. It is very nice to play chess, but it has a big problem with respect to AI.

Miguel

An engine which can evaluate one type of position well may evaluate another type of position badly. This implies that as a series of positions are addressed that gradually stray further away from the “comfort zone” of a particular engine, the evaluation of that engine for those positions can be expected to become less reliable.

That implies to me that, if you put Houdini, or any other engine onto a position that is sufficiently alien, then the evaluation will become virtually useless. I was assuming that, as the positions that HGM was showing Houdini were totally alien to a normal game of chess, the Houdini evaluation would not cope with them particularly well.

I suspect that I am not the only layman who might make those assumptions. If you would explain to me where they break down, I would be grateful.

The whole point of intelligence (artificial or natural) is to adjust to unexpected situations. If you can't, you are not passing the test. There is no such thing as "alien" positions form the point of view of AI. In fact, that is the whole point! we (chess players) are accustom to see some set of positions and most engines are good at that. However, HGM shows the problems they have to make a very simple adjustment, which is understanding material. If the engines fail to make such small and simple adjustment, they fail in regards to an AI experiment.

This is all within the rules of chess, which means that an AI scientist will find them all valid. In fact, those will be more interesting, because a human can easily understand that all the preconceive notions (Q = 9, N = 3) all of the sudden do not apply anymore.

Miguel

[K I Hyams]

The whole point of intelligence (artificial or natural) is to adjust to unexpected situations. If you can't, you are not passing the test. There is no such thing as "alien" positions form the point of view of AI. In fact, that is the whole point! we (chess players) are accustom to see some set of positions and most engines are good at that. However, HGM shows the problems they have to make a very simple adjustment, which is understanding material. If the engines fail to make such small and simple adjustment, they fail in regards to an AI experiment.

This is all within the rules of chess, which means that an AI scientist will find them all valid. In fact, those will be more interesting, because a human can easily understand that all the preconceive notions (Q = 9, N = 3) all of the sudden do not apply anymore.

Miguel

Thank you for your prompt reply. My initial reaction is that we are talking about semantics here. There are many components of an evaluation and a layman would see the dynamic reassessment of values as a normal part of the evaluation function. You refer to it as “AI” I refer to it as “a part of the eval”. Apart from that, I can’t see much difference.

I infer that the AI is more likely to break down in extreme situations. If that is correct, it would seem to be in line with my assessment that the evaluation is more likely to break down in extreme situations.

[Uri Blass]

It seems that QueeNy engine did a stupid mistake in the last loss of it.

It had a drawn position but with queen against 3 knights it sacrificed the queen for 2 knights for no reason.

[Event "?"]
[Site "?"]
[Date "2011.01.19"]
[Round "?"]
[White "New game"]
[Black "Houdini 1.5 w32"]
[Result "0-1"]
[Annotator ",agur"]
[SetUp "1"]
[FEN "2nnkn2/3ppp2/1nn2nn1/8/8/8/3PPP2/1Q1QK1Q1 w - - 0 1"]
[PlyCount "102"]

{256MB, Om Master Book 2.0.ctg, URI-AMD} 1. e3 Nb7 2. d4 Nbd6 3. Qdb3 Ne6 4.
Qf1 Nbc4 5. Qbxc4 Nxc4 6. Qxc4 Nd6 7. Qcb3 Ned8 8. d5 Na5 9. Q3a2 Ndc4 10. Qf5
Nab7 11. Qxf6 exf6 12. Qxc4 Nd6 13. Qc3 Ke7 14. e4 N8b7 15. Qb4 Nf4 16. Kd2 Na5
17. Ke3 Ng6 18. f4 Nac4+ 19. Kf3 Kf8 20. Qe1 f5 21. e5 Ne4 22. Qb4+ Kg7 23. Qb7
Nh4+ 24. Ke2 Ng2 25. Kf3 Ne1+ 26. Ke2 Nd3 27. Kxd3 Nc5+ 28. Kxc4 Nxb7 29. Kb5
f6 30. e6 Nd6+ 31. Kb6 dxe6 32. dxe6 Kf8 33. e7+ Kxe7 34. Kc5 Ne4+ 35. Kc4 Kd6
36. Kd4 Nc5 37. Ke3 Kd5 38. Kf3 Kd4 39. Kf2 Ke4 40. Kg3 Ne6 41. Kh4 Nxf4 42.
Kg3 Nd3 43. Kg2 f4 44. Kf1 f3 45. Kg1 Ke3 46. Kf1 f2 47. Kg2 Ke2 48. Kg3 f1=Q
49. Kg4 Qf3+ 50. Kh4 Qg2 51. Kh5 Qg5# 0-1

[D]8/1Q1p1pk1/8/3PPp2/2n1nP2/3n4/4K3/8 w - - 0 27 am Kxd3

Most engines will have no problem with this position and houdini evaluates it as 0.00 if white play Kf3

[bhlangonijr]

The whole point of intelligence (artificial or natural) is to adjust to unexpected situations. If you can't, you are not passing the test. There is no such thing as "alien" positions form the point of view of AI. In fact, that is the whole point! we (chess players) are accustom to see some set of positions and most engines are good at that. However, HGM shows the problems they have to make a very simple adjustment, which is understanding material. If the engines fail to make such small and simple adjustment, they fail in regards to an AI experiment.

This is all within the rules of chess, which means that an AI scientist will find them all valid. In fact, those will be more interesting, because a human can easily understand that all the preconceive notions (Q = 9, N = 3) all of the sudden do not apply anymore.
Miguel

The current state-of-art chess engines are almost brute-force searchers. The knowledge is not generalized but rather specialized by summing up a handful of à la Betty Crocker's recipes for evaluating specific things in a chess position, so that the most successful chess engines are those which are doing that well for most cases - or by completely out-searching other engines or even both. HGM's position only proves the point so there is not surprise here.
Undoubtedly computer chess are doing very well nowadays but the question is whether it really is a successful AI agent or not. IMO the answer is NOT.

[rbarreira]

The whole point of intelligence (artificial or natural) is to adjust to unexpected situations. If you can't, you are not passing the test. There is no such thing as "alien" positions form the point of view of AI. In fact, that is the whole point! we (chess players) are accustom to see some set of positions and most engines are good at that. However, HGM shows the problems they have to make a very simple adjustment, which is understanding material. If the engines fail to make such small and simple adjustment, they fail in regards to an AI experiment.

This is all within the rules of chess, which means that an AI scientist will find them all valid. In fact, those will be more interesting, because a human can easily understand that all the preconceive notions (Q = 9, N = 3) all of the sudden do not apply anymore.
Miguel

The current state-of-art chess engines are almost brute-force searchers. The knowledge is not generalized but rather specialized by summing up a handful of à la Betty Crocker's recipes for evaluating specific things in a chess position, so that the most successful chess engines are those which are doing that well for most cases - or by completely out-searching other engines or even both. HGM's position only proves the point so there is not surprise here.
Undoubtedly computer chess are doing very well nowadays but the question is whether it really is a successful AI agent or not. IMO the answer is NOT.

It is a successful AI agent in the sense that it plays chess really well (defining "playing chess well" as playing a full game with normal chess rules, which would hardly end up in a 9 knights, 3 queens position with good play from the AI agent).

But then again, for many people AI is always defined as "that which computers can't do well yet", in which case no computer program can ever be successful at AI (moving goalposts and all that).

[bhlangonijr]

It is a successful AI agent in the sense that it plays chess really well (defining "playing chess well" as playing a full game with normal chess rules, which would hardly end up in a 9 knights, 3 queens position with good play from the AI agent).

But then again, for many people AI is always defined as "that which computers can't do well yet", in which case no computer program can ever be successful at AI (moving goalposts and all that).

I completely disagree and I think "that which computers can't do well yet" is a very lousy definition for AI. I'll quote something which IMO captures better what AI should be: "..where an intelligent agent is a system that perceives its environment and takes actions that maximize its chances of success." - John MacCarth. There must be some sort of "conscience" within the AI agent so that it can dynamically change its behaviour to better adapt to different situations. That's exactly what the current approaches to implement a top chess engine are NOT considering. As Miguel nicely put it is overfitted. Computers do a lot of things very well and it doesn't mean they have intelligent behaviour because of that.