Using test suites -- when to say the position is solved?

[Fguy64]

Note that if a sufficiently fast forced mate (or loss) is found, the program should exit the test early for that position.

I test a set of positions iteratively (starting at one second per position) with each test allowing twice the time than the prior test run. Problems solved in one run are not tried in succeeding runs.

I've been thinking about this thread as I worked my way through the WAC positions.

Unless I misunderstand the intent of the bolded remark, is there not an issue here whereby the program stops after finding a fast enough forced mate, but overlooks an even faster mate because it stopped too soon?

As an example, there was a position whereby my program found a mate in four, but as it played out the line, it never reached the mate, because at each turn it kept finding mates in 4 before it found the mate in 2 move that was part of the original line. Or something very much like that

Naturally a properly designed program that "saves its analysis" won't have this problem, but that rather begs the question IMO. Anyways, it's just an example of how an engine can find a forcing line, yet be unable to drive home the point.

Anyways, until I am able to do it properly, I just added a routine whereby the engine would reduce the ply by 2 whenever it played a move in a line that forced checkmate.

[plattyaj]

Note that if a sufficiently fast forced mate (or loss) is found, the program should exit the test early for that position.

I test a set of positions iteratively (starting at one second per position) with each test allowing twice the time than the prior test run. Problems solved in one run are not tried in succeeding runs.

I've been thinking about this thread as I worked my way through the WAC positions.

Unless I misunderstand the intent of the bolded remark, is there not an issue here whereby the program stops after finding a fast enough forced mate, but overlooks an even faster mate because it stopped too soon?

Yes. Particularly as the search "improves" it can extend deeper in places and reduce searching in others so that you can easily find the longer mate when searching to a lower depth.

What Schola does, and I assume it's pretty common, is to carry on searching for a quicker mate until it's found a mate that is 4 plies less than the current depth it's searching too. In analysis mode it will actually carry on forever.

Andy.

[Fguy64]

Note that if a sufficiently fast forced mate (or loss) is found, the program should exit the test early for that position.

I test a set of positions iteratively (starting at one second per position) with each test allowing twice the time than the prior test run. Problems solved in one run are not tried in succeeding runs.

I've been thinking about this thread as I worked my way through the WAC positions.

Unless I misunderstand the intent of the bolded remark, is there not an issue here whereby the program stops after finding a fast enough forced mate, but overlooks an even faster mate because it stopped too soon?

Yes. Particularly as the search "improves" it can extend deeper in places and reduce searching in others so that you can easily find the longer mate when searching to a lower depth.

What Schola does, and I assume it's pretty common, is to carry on searching for a quicker mate until it's found a mate that is 4 plies less than the current depth it's searching too. In analysis mode it will actually carry on forever.

Andy.

hmm, I'm not sure I follow. Answer me this first. When schola embarks upon a mating line, how does it "remember the mate" so that in subsequent turns it can take advantage of analysis that has already been conducted.

[plattyaj]

hmm, I'm not sure I follow. Answer me this first. When schola embarks upon a mating line, how does it "remember the mate" so that in subsequent turns it can take advantage of analysis that has already been conducted.

If we are talking about a single search, it's just the normal iterative deepening. It does a search to ply 1, gets the best move; now it does a search to ply 2, gets the best move and the best reply - there's the PV. Each time the best move at the root is tried first for the next iteration.

For mates, what might happen is that when I'm searching to ply 4 I find a mate in 4 (8 half-moves) that has a lot of checks in it (I'm extending the search for a check) and that becomes the best move. But at ply 6 I find there's actually a mate in three that I hadn't seen earlier so that is now the best move.

If you are talking about a sequence of moves all with a search (search position, play, opponent plays, search position ...), the only thing carried over is the hash table.

Andy.

[Fguy64]

hmm, I'm not sure I follow. Answer me this first. When schola embarks upon a mating line, how does it "remember the mate" so that in subsequent turns it can take advantage of analysis that has already been conducted.

If we are talking about a single search, it's just the normal iterative deepening. It does a search to ply 1, gets the best move; now it does a search to ply 2, gets the best move and the best reply - there's the PV. Each time the best move at the root is tried first for the next iteration.

For mates, what might happen is that when I'm searching to ply 4 I find a mate in 4 (8 half-moves) that has a lot of checks in it (I'm extending the search for a check) and that becomes the best move. But at ply 6 I find there's actually a mate in three that I hadn't seen earlier so that is now the best move.

If you are talking about a sequence of moves all with a search (search position, play, opponent plays, search position ...), the only thing carried over is the hash table.

Andy.

OK, I see what you mean more or less. My system is really basic, so far I don't deal with iterative deepening or search extensions. As for hash tables, well I understand the purpose, and I've dabbled with some really basic stuff, but for me a lot of the available literature, including that at the chess programming wiki, is too deep, so it's gonna take some time to get my head this stuff.

[Don]

I don't use problems too much, but some, mostly as a kind of quick sanity check.

Larry Kaufman and I have found that it's more reliable to count the time to iteration completion. For instance if it finds the solution on the 7th iteration, we wait until depth 7 is complete. I think it just takes some of the random noise out of the time.

It's expensive to test if it keeps a solution. That means you cannot run tough problems (I cannot let it think 10 minutes to see if a problem it solved in 30 seconds is kept for several iterations in a row.)

I like the kind of problems that have only 1 solution, are completely unambiguous and the program is not likely to find the right move for the wrong reason (by accident.) If the program finds the solution, then it clearly understands the position and it is not as likely to change it's mind.

I would like to find a good "pure" set like that. I have not looked too hard, but every set I have generally has some of these ambiguous positions.