New Scorpio bitbase files

[Daniel Shawul]

Thanks Denis! Lets wait for some more tests before we make them
official. I tested with Toga and Bright here which worked successfully.
If no one comes up with more bugs in a weeks time, I will just take your
compiles (if you don't mind) and post it on my web page.

regards,
Daniel

[Daniel Shawul]

Take this position for MES.epd for example

[d] k7/5K2/8/8/7p/p6P/P7/8 w - - 0 1

We have a 6 man endgame tb here while we have only 5 men egbbs or tbs.
The position "transforms" to a won position once the king captures the pawn at a3. Scorpio analyzes this position as a won

Code: Select all

FEN: k7/5K2/8/8/7p/p6P/P7/8 w - - 0 1 

Scorpio:
   2	00:00	          13	13	+1.18	Kf7-e6 Ka8-b7
   2	00:00	          27	27	+1.18	Kf7-e6 Ka8-b7
   3	00:00	          51	51	+1.28	Kf7-e6 Ka8-b7 Ke6-e5
   3	00:00	         112	112	+1.28	Kf7-e6 Ka8-b7 Ke6-e5
   4	00:00	         137	137	+0.78	Kf7-e6
   4	00:00	         252	252	+1.44	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4
   4	00:00	         303	303	+1.44	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4
   5	00:00	         419	419	+1.44	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4
   5	00:00	         528	528	+1.44	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4
   6	00:00	         680	680	+0.82	Kf7-f6
   6	00:00	         960	960	+0.82	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4 Kc6-d5
   6	00:00	         996	996	+0.82	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4 Kc6-d5
   7	00:00	       1.103	1.103	+1.39	Kf7-f6
   7	00:00	       1.358	1.358	+1.41	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4 Kc6-d5 Kh4-g4
   7	00:00	       1.576	1.576	+1.41	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4 Kc6-d5 Kh4-g4
   8	00:00	       1.868	1.868	+0.82	Kf7-f6
   8	00:00	       3.223	3.223	+0.39	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5-h6 Kc6-d5 Kh6-h5 Kd5-e4 Kh5xh4
   8	00:00	       4.301	4.301	+0.48	Kf7-g7 Ka8-b7 Kg7-f6 Kb7-c6 Kf6-e6 Kc6-c7 Ke6-e7
   8	00:00	       4.335	4.335	+0.48	Kf7-g7 Ka8-b7 Kg7-f6 Kb7-c6 Kf6-e6 Kc6-c7 Ke6-e7
   9	00:00	       5.265	5.265	+0.39	Kf7-g7 Ka8-b7 Kg7-f6 Kb7-c7 Kf6-g5 Kc7-c6 Kg5-h6
   9	00:00	       5.821	5.821	+0.98	Kf7-e6
   9	00:00	       6.598	6.598	+0.98	Kf7-e6 Ka8-b7 Ke6-f5 Kb7-c6 Kf5-g5 Kc6-d5 Kg5xh4 Kd5-e4 Kh4-g4
   9	00:00	       6.699	6.699	+0.98	Kf7-e6 Ka8-b7 Ke6-f5 Kb7-c6 Kf5-g5 Kc6-d5 Kg5xh4 Kd5-e4 Kh4-g4
  10	00:00	       7.704	7.704	+0.98	Kf7-e6 Ka8-b8 Ke6-f5 Kb8-c7 Kf5-g4 Kc7-d6 Kg4xh4 Kd6-e5 Kh4-g4 Ke5-e4
  10	00:00	       8.522	8.522	+0.98	Kf7-e6 Ka8-b8 Ke6-f5 Kb8-c7 Kf5-g4 Kc7-d6 Kg4xh4 Kd6-e5 Kh4-g4 Ke5-e4
  11	00:00	       9.907	990.700	+0.94	Kf7-e6 Ka8-b7 Ke6-f5 Kb7-c7 Kf5-g4 Kc7-d6 Kg4-h5 Kd6-e5 Kh5xh4 Ke5-f4 Kh4-h5
  11	00:00	      10.612	1.061.200	+0.94	Kf7-e6 Ka8-b7 Ke6-f5 Kb7-c7 Kf5-g4 Kc7-d6 Kg4-h5 Kd6-e5 Kh5xh4 Ke5-f4 Kh4-h5
  12	00:00	      12.882	1.288.200	+0.46	Kf7-e6 Ka8-b8 Ke6-f5 Kb8-c7 Kf5-g4 Kc7-d6 Kg4-h5 Kd6-e5 Kh5xh4 Ke5-f4 Kh4-h5 Kf4-e5
  12	00:00	      14.607	1.460.700	+0.46	Kf7-e6 Ka8-b8 Ke6-f5 Kb8-c7 Kf5-g4 Kc7-d6 Kg4-h5 Kd6-e5 Kh5xh4 Ke5-f4 Kh4-h5 Kf4-e5
  13	00:00	      25.710	2.571.000	+0.94	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c4 Kc8-b7 Kc4-b3 Kb7-c6 Kb3xa3 Kc6-d5 Ka3-b4 Kd5-e4 Kb4-c5
  13	00:00	      26.531	2.653.100	+0.94	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c4 Kc8-b7 Kc4-b3 Kb7-c6 Kb3xa3 Kc6-d5 Ka3-b4 Kd5-e4 Kb4-c5
  14	00:00	      28.478	949.266	+46.52	Kf7-e6
  15	00:00	      31.832	1.061.066	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  15	00:00	      33.333	1.111.100	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  16	00:00	      35.147	1.171.566	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  16	00:00	      38.084	1.269.466	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  17	00:00	      41.163	1.029.075	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  17	00:00	      42.860	1.071.500	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  18	00:00	      44.811	1.120.275	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  18	00:00	      48.348	1.208.700	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  19	00:00	      53.559	892.650	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  19	00:00	      56.782	946.366	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  20	00:00	      58.557	975.950	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  20	00:00	      61.831	1.030.516	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  21	00:00	      67.808	753.422	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  21	00:00	      71.091	789.900	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  22	00:00	      73.762	670.563	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  22	00:00	      78.583	714.390	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  23	00:00	      85.877	780.700	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  23	00:00	      89.526	813.872	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  24	00:00	      91.937	612.913	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  24	00:00	      97.284	648.560	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3

In this case there are no egtbs to switch to for making progress. But still scorpio can make progress and finish the mate with the method i mentioned in the other thread. But obviously we can't discard any root moves here. BTW I don't think that searching only winning moves at the root helps at all for bitbases. The alpha-beta search already does that for us and we can say they are almost not searched there. If you have two or more wins you still have to differentiate between the two. I think you are only thinking of EGTBs where the mate score has the "N ply" in it. The egbbs don't have the ply information in them which is causing the problem. For draws and losss I immediately return the score without searching just for fair play. You can make this difficult hoping the other engine does a mistake.

Switching to EGTBs once we reach the 5 men endgame solves some of the problem but does not solve all our problems if you use bitbases.
I tried to look into crafty and the only reason I can think of that you had it is for cases I mentioned above i.e. to make crafty play "better moves" even when it knows the position is draw.

Daniel

[bob]

I don't see how it helps.

We start off in a 5-piece ending, so bitbase probes are 100%.

I thought that you said the swindle mode works by excluding the non-winning _root moves_ and searching only the winning moves. For that to work the position at the root should be in the egbb in the first place? So I don't see how that works if I have a 7 man position at the root (while having only 5 men bitbases). The search can still return mate scores because of exchanges. You would have to explain how the swindle mode works before we go on...

What I do is probe the bitbases in case we have captures/promotions/pawn pushes immediately even in the first 2/3rd of the search tree. Now if that doesn't cutoff the tree and we reach the "pseudo horizon" (2/3 rd boundary), we do a probe and then adjust the score by the ply. Pretty much like mate scores. If we just returned -MATE then we won't be able to progress. So I do -WIN_SCORE + factor * (PLY) that makes sure mates at shallower depth are prefered. There are some issues with the Hash table that should be handled but this pretty much worked for me for some time... Forget the KQPKQ case that keeps to be posted here again and again.

Also scores are adjusted to make better progress in specially difficult positonss liek KbnK which requires different king piece square table.
For the general case I have this
score = WIN_SCORE
- 10 * distance(SQ6488(w_king),SQ6488(b_king))
+ (5 - all_c) * 1000;


Daniel

Let's back up to the beginning.

(1) > 5 pieces otb at the root position. Here a normal search is used, including bitbase accesses _everywhere_ in the tree, which will guide you to positions that are won, assuming there are some. It makes no sense to not probe in the first 2/3 of the tree here, because the _only_ score you get comes from either the bitbase win/lose/draw score, or for lines that do not contain any captures, you go with whatever the eval backs up. But the assumption here is that you are going to force the game to a won position according to bitbases. And you continue to search in this way until,

(2) 5 pieces (or less) otb at the root position. Here any search that relies on bitbases is going to blunder a win into a draw. It won't lose of course, but going from a win to a draw by 50 move rule is bad enough. And here, searching the first 2/3 of the tree without bitbase probes gets you nothing, because at the very first ply you do choose to probe, you get a hit, and you get no evaluation other than "win/lose/draw". We already know we can force our way to a "win". But we also know that all wins are equal and ignore the 50 move rule. So you have to _hope_ that you can see the 50 move rule draws before you are in so deep you can't get out of the 50-move rule draw by any possible move path.

So what you described does not solve the problem. It still leaves you blind to "winning pathways" that will run into the 50-move rule draw because your "winning pathways" have no concept of "making progress". Yes you might, in the first 2/3 of the search, recognize a repetition or 50 move draw, and avoid those. Good. But you won't recognize a lot of them in that way, until it is too late. And you end up with a draw...

[Daniel Shawul]

5 pieces (or less) otb at the root position. Here any search that relies on bitbases is going to blunder a win into a draw. It won't lose of course, but going from a win to a draw by 50 move rule is bad enough. And here, searching the first 2/3 of the tree without bitbase probes gets you nothing, because at the very first ply you do choose to probe, you get a hit, and you get no evaluation other than "win/lose/draw". We already know we can force our way to a "win". But we also know that all wins are equal and ignore the 50 move rule. So you have to _hope_ that you can see the 50 move rule draws before you are in so deep you can't get out of the 50-move rule draw by any possible move path.

Here is something you probably missed. I do immediate cutoffs when I have captures/promotions/pawn pushes (hoping progress is made).For the rest I let it search for 2/3rd of the positions before I probe the bitbases and cutoff the tree. Now all the positions at the 2/3rd border line have exactly the same number of pieces as the root with the only difference that the pieces are shuffled. I still try to differentiate between these positions by evaluating the position. Questions like 'Are the pieces closer to the loosing king now?" etc.. So this _does_ work for most. If you just probe the bitbases at the root you are not giving it any chance to see captures, promotions, and shuffling of pieces which lead to better evaluation. So just returning the score closer to the root does not work at all for peice <=5. I have tried this and the optimum that I chose is 2/3rd , therefore this number does have some merits on the progress.

I still did not understand the swindle mode fully. Have you read my other reply btw?

Daniel

[bob]

5 pieces (or less) otb at the root position. Here any search that relies on bitbases is going to blunder a win into a draw. It won't lose of course, but going from a win to a draw by 50 move rule is bad enough. And here, searching the first 2/3 of the tree without bitbase probes gets you nothing, because at the very first ply you do choose to probe, you get a hit, and you get no evaluation other than "win/lose/draw". We already know we can force our way to a "win". But we also know that all wins are equal and ignore the 50 move rule. So you have to _hope_ that you can see the 50 move rule draws before you are in so deep you can't get out of the 50-move rule draw by any possible move path.

Here is something you probably missed. I do immediate cutoffs when I have captures/promotions/pawn pushes (hoping progress is made).For the rest I let it search for 2/3rd of the positions before I probe the bitbases and cutoff the tree. Now all the positions at the 2/3rd border line have exactly the same number of pieces as the root with the only difference that the pieces are shuffled. I still try to differentiate between these positions by evaluating the position. Questions like 'Are the pieces closer to the loosing king now?" etc.. So this _does_ work for most. If you just probe the bitbases at the root you are not giving it any chance to see captures, promotions, and shuffling of pieces which lead to better evaluation. So just returning the score closer to the root does not work at all for peice <=5. I have tried this and the optimum that I chose is 2/3rd , therefore this number does have some merits on the progress.

I still did not understand the swindle mode fully. Have you read my other reply btw?

Daniel

Swindle mode applies to draws, but it works for this case as well.

If you reach a 5 piece position at the root, where the bitbase result is "won", then you generate all legal moves. You make each move and then probe the bitbase for the resulting position which should now say "lost" since you have flipped sides. However, some of these moves will possibly say "won" since in a won position there might be losing moves as well. For any move where the result is anything but lost, you just cull those moves. So you are left with moves that take the opponent to a lost position, no others remain.

Now disable bitbases and just search normally. You probably have some sort of "mop-up evaluation" that recognizes who is losing and who is winning, and tries to drive the losing side to the edge of the board first, and then to the correct corner if that is needed (KBN vs K for example). You continue to only search winning moves at the root and let your evaluation pick the one that appears to make the most progress. This will work in most cases, but difficult endings like KRP vs KR you need some small but specific evaluation terms to understand how to win or draw such positions dependin on which you want to do.

For draws, I do this same thing, culling all moves that lead to losses, and then do a search. This avoids the ugly case where you are in a drawn ending, and you just give up the pawn since that is still a drawn position. This "swindle mode" will use the normal search (no tables used anywhere) so that the program tries to preserve the pawn and advance it as well, and if the opponent makes a mistake, we still check at the root and if we see a forced win we go for it, otherwise we search to make the opponent work as hard as possible to hold the draw and hope he makes a mistake.

Both ideas are the same here, in terms of code, although the desired result is different.

[bob]

Take this position for MES.epd for example

[d] k7/5K2/8/8/7p/p6P/P7/8 w - - 0 1

We have a 6 man endgame tb here while we have only 5 men egbbs or tbs.
The position "transforms" to a won position once the king captures the pawn at a3. Scorpio analyzes this position as a won

Code: Select all

FEN: k7/5K2/8/8/7p/p6P/P7/8 w - - 0 1 

Scorpio:
   2	00:00	          13	13	+1.18	Kf7-e6 Ka8-b7
   2	00:00	          27	27	+1.18	Kf7-e6 Ka8-b7
   3	00:00	          51	51	+1.28	Kf7-e6 Ka8-b7 Ke6-e5
   3	00:00	         112	112	+1.28	Kf7-e6 Ka8-b7 Ke6-e5
   4	00:00	         137	137	+0.78	Kf7-e6
   4	00:00	         252	252	+1.44	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4
   4	00:00	         303	303	+1.44	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4
   5	00:00	         419	419	+1.44	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4
   5	00:00	         528	528	+1.44	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4
   6	00:00	         680	680	+0.82	Kf7-f6
   6	00:00	         960	960	+0.82	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4 Kc6-d5
   6	00:00	         996	996	+0.82	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4 Kc6-d5
   7	00:00	       1.103	1.103	+1.39	Kf7-f6
   7	00:00	       1.358	1.358	+1.41	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4 Kc6-d5 Kh4-g4
   7	00:00	       1.576	1.576	+1.41	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5xh4 Kc6-d5 Kh4-g4
   8	00:00	       1.868	1.868	+0.82	Kf7-f6
   8	00:00	       3.223	3.223	+0.39	Kf7-f6 Ka8-b7 Kf6-g5 Kb7-c6 Kg5-h6 Kc6-d5 Kh6-h5 Kd5-e4 Kh5xh4
   8	00:00	       4.301	4.301	+0.48	Kf7-g7 Ka8-b7 Kg7-f6 Kb7-c6 Kf6-e6 Kc6-c7 Ke6-e7
   8	00:00	       4.335	4.335	+0.48	Kf7-g7 Ka8-b7 Kg7-f6 Kb7-c6 Kf6-e6 Kc6-c7 Ke6-e7
   9	00:00	       5.265	5.265	+0.39	Kf7-g7 Ka8-b7 Kg7-f6 Kb7-c7 Kf6-g5 Kc7-c6 Kg5-h6
   9	00:00	       5.821	5.821	+0.98	Kf7-e6
   9	00:00	       6.598	6.598	+0.98	Kf7-e6 Ka8-b7 Ke6-f5 Kb7-c6 Kf5-g5 Kc6-d5 Kg5xh4 Kd5-e4 Kh4-g4
   9	00:00	       6.699	6.699	+0.98	Kf7-e6 Ka8-b7 Ke6-f5 Kb7-c6 Kf5-g5 Kc6-d5 Kg5xh4 Kd5-e4 Kh4-g4
  10	00:00	       7.704	7.704	+0.98	Kf7-e6 Ka8-b8 Ke6-f5 Kb8-c7 Kf5-g4 Kc7-d6 Kg4xh4 Kd6-e5 Kh4-g4 Ke5-e4
  10	00:00	       8.522	8.522	+0.98	Kf7-e6 Ka8-b8 Ke6-f5 Kb8-c7 Kf5-g4 Kc7-d6 Kg4xh4 Kd6-e5 Kh4-g4 Ke5-e4
  11	00:00	       9.907	990.700	+0.94	Kf7-e6 Ka8-b7 Ke6-f5 Kb7-c7 Kf5-g4 Kc7-d6 Kg4-h5 Kd6-e5 Kh5xh4 Ke5-f4 Kh4-h5
  11	00:00	      10.612	1.061.200	+0.94	Kf7-e6 Ka8-b7 Ke6-f5 Kb7-c7 Kf5-g4 Kc7-d6 Kg4-h5 Kd6-e5 Kh5xh4 Ke5-f4 Kh4-h5
  12	00:00	      12.882	1.288.200	+0.46	Kf7-e6 Ka8-b8 Ke6-f5 Kb8-c7 Kf5-g4 Kc7-d6 Kg4-h5 Kd6-e5 Kh5xh4 Ke5-f4 Kh4-h5 Kf4-e5
  12	00:00	      14.607	1.460.700	+0.46	Kf7-e6 Ka8-b8 Ke6-f5 Kb8-c7 Kf5-g4 Kc7-d6 Kg4-h5 Kd6-e5 Kh5xh4 Ke5-f4 Kh4-h5 Kf4-e5
  13	00:00	      25.710	2.571.000	+0.94	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c4 Kc8-b7 Kc4-b3 Kb7-c6 Kb3xa3 Kc6-d5 Ka3-b4 Kd5-e4 Kb4-c5
  13	00:00	      26.531	2.653.100	+0.94	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c4 Kc8-b7 Kc4-b3 Kb7-c6 Kb3xa3 Kc6-d5 Ka3-b4 Kd5-e4 Kb4-c5
  14	00:00	      28.478	949.266	+46.52	Kf7-e6
  15	00:00	      31.832	1.061.066	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  15	00:00	      33.333	1.111.100	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  16	00:00	      35.147	1.171.566	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  16	00:00	      38.084	1.269.466	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  17	00:00	      41.163	1.029.075	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  17	00:00	      42.860	1.071.500	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  18	00:00	      44.811	1.120.275	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  18	00:00	      48.348	1.208.700	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  19	00:00	      53.559	892.650	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  19	00:00	      56.782	946.366	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  20	00:00	      58.557	975.950	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  20	00:00	      61.831	1.030.516	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  21	00:00	      67.808	753.422	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  21	00:00	      71.091	789.900	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  22	00:00	      73.762	670.563	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  22	00:00	      78.583	714.390	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  23	00:00	      85.877	780.700	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  23	00:00	      89.526	813.872	+46.52	Kf7-e6 Ka8-b7 Ke6-d5 Kb7-c8 Kd5-c5 Kc8-d7 Kc5-b4 Kd7-e6 Kb4xa3
  24	00:00	      91.937	612.913	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3
  24	00:00	      97.284	648.560	+46.52	Kf7-e6 Ka8-b8 Ke6-d5 Kb8-c7 Kd5-c5 Kc7-d7 Kc5-b4 Kd7-e6 Kb4xa3

In this case there are no egtbs to switch to for making progress. But still scorpio can make progress and finish the mate with the method i mentioned in the other thread. But obviously we can't discard any root moves here. BTW I don't think that searching only winning moves at the root helps at all for bitbases. The alpha-beta search already does that for us and we can say they are almost not searched there. If you have two or more wins you still have to differentiate between the two. I think you are only thinking of EGTBs where the mate score has the "N ply" in it. The egbbs don't have the ply information in them which is causing the problem. For draws and losss I immediately return the score without searching just for fair play. You can make this difficult hoping the other engine does a mistake.

Switching to EGTBs once we reach the 5 men endgame solves some of the problem but does not solve all our problems if you use bitbases.
I tried to look into crafty and the only reason I can think of that you had it is for cases I mentioned above i.e. to make crafty play "better moves" even when it knows the position is draw.

Daniel

searching only winning moves means your bitbase-less search won't make a mistake at the root and convert the win to a draw or loss, while letting your normal search and evaluation make the progress.

A better example might be KNN vs KP. You can trivially drag that out excessively and draw it if you make a single mistake. Ditto for KQ vs KR as another example. After playing 30 moves into the 5-piece position, you might have two forced win moves where one is a mate in 15, the other is a mate in 25. The latter will end up drawing... The former will win. Hopefully your evaluation has enough info that it can take _most_ won positions and actually win them, particularly when you eliminate the moves that lose or draw at the root...

[Daniel Shawul]

Swindle mode applies to draws, but it works for this case as well.

If you reach a 5 piece position at the root, where the bitbase result is "won", then you generate all legal moves. You make each move and then probe the bitbase for the resulting position which should now say "lost" since you have flipped sides. However, some of these moves will possibly say "won" since in a won position there might be losing moves as well. For any move where the result is anything but lost, you just cull those moves. So you are left with moves that take the opponent to a lost position, no others remain.

I don't think this helps us to advance in anyway. Whether the loosing root moves are there
or not, the engine will not waste much time with them (becasue of alpha-beta) neither plays them.
BTW I just realized that I do cull the loosing root moves in the recent versions. This was a side
effect of the code I wrote to make scorpio move immediately when it is loosing and drawing (fair
play for the opponent) So when some are drawing moves and some winning only the later are searched.
Before I did this all are searched with no side effect at all. The only problem with using bitbases is
going beyond the 50-move rule limit which should not have been there in the first place.
Ofcourse when we have 6 or more pieces at the root, there are no moves to cull at all, so obviously
no help from that.

Now disable bitbases and just search normally. You probably have some sort of "mop-up evaluation" that recognizes who is losing and who is winning, and tries to drive the losing side to the edge of the board first, and then to the correct corner if that is needed (KBN vs K for example). You continue to only search winning moves at the root and let your evaluation pick the one that appears to make the most progress. This will work in most cases, but difficult endings like KRP vs KR you need some small but specific evaluation terms to understand how to win or draw such positions dependin on which you want to do.

Ok disabling the bitbases and searching to the full depth is clearly better if you have <= 5 men at the root
because you invest the whole search depth for that. But if you have 6,7 or more pieces at the root where you actually need
the bitbases to see the positon is actually a win, you have to probe somewhere! Otherwise first it
says score is +46.52 for the 6men then you disable the bitbases for the next move you get +9.00 your
search/eval is good enough. You might even completly loose the plot and play some random move (I saw it
happen for kppkp cases). Disabling them causes search instability or in the worst case might loose you a possible
win. Thats why I chose a general scheme of probing upto a certain depth (2/3rd in this case). This depth is adjustable
and can be set to the whole search depth, which is similar to disabling it all in all. Infact this may be better
because in your case you use the regualr evaluation function, but in my case *the actual score from the bitbases* ,
rather than material based evaluation, is modified by some factors and returned.

For draws, I do this same thing, culling all moves that lead to losses, and then do a search. This avoids the ugly case where you are in a drawn ending, and you just give up the pawn since that is still a drawn position. This "swindle mode" will use the normal search (no tables used anywhere) so that the program tries to preserve the pawn and advance it as well, and if the opponent makes a mistake, we still check at the root and if we see a forced win we go for it, otherwise we search to make the opponent work as hard as possible to hold the draw and hope he makes a mistake.

Both ideas are the same here, in terms of code, although the desired result is different.

From what I understood of your swindle mode code, you just modify the draw score by 1 to pretend it is not a draw and
let the search find winning paths against an opponent which does not have egtb support. That is similar to what I do
except that I do more elaborate evaluations to progress in winning positions. For draws I don't even care to search them
just for fair play. I except other engines to just resign, rather than make me work, for positions like a won KQPKQ position even if I use bitbases

Daniel

[Dirt]

The only problem with using bitbases is going beyond the 50-move rule limit which should not have been there in the first place.

I'm guessing your bitbases pay no attention to the fifty move rule. That is, if a position is a forced mate with a minimum of sixty moves to conversion you treat it as a won position. Is this correct?

[Daniel Shawul]

searching only winning moves means your bitbase-less search won't make a mistake at the root and convert the win to a draw or loss, while letting your normal search and evaluation make the progress.

Right, so in those cases that a wrong root move is backed up(found out with the bitbase-less search) it means the search needs some help from the bitbases not just at the root but down the search tree also. That is basically what I am trying to do. Let the search do its stuff for some search depth (2/3rd) then cutoff from that point onwards. If I make probe_depth less it won't be able to make progress. You have to probe somewhere close to the leaves to find the correct move at the root!! And for bitbases on disk (5 men are big) probing when the remaining depth left is 2 or 3 is bad. That is why I chose to not use the whole search depth (I really forgot to mention this until you brought up this case!) . For 4 men infact I do probes down the queiscene search also but not for 5 mens.

A better example might be KNN vs KP. You can trivially drag that out excessively and draw it if you make a single mistake. Ditto for KQ vs KR as another example. After playing 30 moves into the 5-piece position, you might have two forced win moves where one is a mate in 15, the other is a mate in 25. The latter will end up drawing... The former will win. Hopefully your evaluation has enough info that it can take _most_ won positions and actually win them, particularly when you eliminate the moves that lose or draw at the root...

Note that I do probes inside search tree (no bitbase-less search for me) to effect mates with 6,7 men at the root. You don't have that problem because you are using EGTB score which has the _Mate in N_ value. Not so for bitbases, all you can do is guess the "N" value by how fast it converts to other postions etc... My search always backs up a correct root move that is why it makes no difference for me to cull moves at the root. But I understand you could have wrong moves there in your case.

[Daniel Shawul]

Yes