SjaakII 1.0 RC1

[myfish]

building now.

[Evert]

Windows binaries are up too.

[hgm]

Very nice! I configured Euro-Shogi with mandatory promotion now, which was the only imperfection left. RC5 also seems significantly stronger than earlier versions. Shokidoki now really has to work hard to beat it (and does not always manage that), in Euro-Shogi.

I encountered no problems so far. In the following game it is suddenly checkmated while both engines agreed it was significantly ahead, but I guess that is just lack of depth.

Code: Select all

[Event "Computer Chess Game"]
[Site "ONTWIKKELLAPTOP"]
[Date "2015.02.26"]
[Round "5"]
[White "Sjaak II 1.0 RC5"]
[Black "Shokidoki euro"]
[Result "0-1"]
[TimeControl "40/300"]
[Variant "euroshogi"]
[VariantMen "P:fW;N:sWffN;B:B;R:R;G:WfF;+P:WfF;+N:WfF;+B:BW;+R:RF;K:K;+p:WfF;+n:WfF;+b:BW;+r:RF"]
[FEN "1nbgkgn1/1r4b1/pppppppp/8/8/PPPPPPPP/1B4R1/1NGKGBN1[-] w 0 1"]
[SetUp "1"]

{--------------
. n b g k g n .
. r . . . . b .
p p p p p p p p
. . . . . . . .
. . . . . . . .
P P P P P P P P
. B . . . . R .
. N G K G B N .
white to play
--------------}
1. c4 {+0.08/12} Ke7 {+0.56/13 4} 2. f4 {+0.45/13 5} Gf7 {+0.72/12 9} 3.
Rf2 {+0.65/13 8} Nf8 {+0.84/11 3} 4. Bg2 {+0.75/13 4} Ne8 {+0.80/12 7} 5.
Bd4 {+1.00/12 4} b5 {+2.00/11 3} 6. Kc2 {+1.03/12 4} b4 {+1.80/12 5} 7.
bxb4 {+2.07/11 8} Rxb4 {+1.08/11 10} 8. Ba7+ {+2.65/10 6} g5 {+1.20/10 4}
9. P@b6 {+4.22/9 5} Rxc4 {-1.60/10 6} 10. Kb3 {+6.07/10 6} g4 {-2.36/10 5}
11. gxg4 {+7.36/11 11} Rxc1+ {+1.72/11 4} 12. Nxc1 {+4.63/10 8} G@g3
{+0.80/10 7} 13. +Bxb8 {+4.16/9 7} Gxg2= {+0.24/9 4} 14. N@d5 {+4.28/8 6}
dxd5 {-0.16/10 14} 15. Rxg2 {+4.67/9 10} B@a5 {+0.08/9 6} 16. R@a7
{+5.22/8 8} Bd7 {-0.36/9 10} 17. Gd2 {+5.46/9 9} P@g3 {+1.44/8 4} 18. Re2
{+5.74/8 5} N@c8 {+0.44/8 13} 19. +Bb7 {+6.40/9 5} g2+ {-0.72/9 6} 20. Rxg2
{+6.35/9 6} Bxb6 {-1.36/9 3} 21. Ra8+ {+8.06/9 5} Gc7 {-2.00/9 4} 22. +Bxa6
{+8.72/9 5} d4 {-3.32/8 4} 23. G@h7 {+9.52/9 8} dxd3+ {-3.68/9 5} 24. +Bxd3
{+9.14/8 7} P@d4 {-2.12/10 6} 25. +Bb1 {+10.28/10 9} c5 {-1.60/9 4} 26.
Gxg7 {+8.39/9 9} Gxg7 {-1.80/9 4} 27. B@e4 {+7.72/9 9} e5 {+0.16/8 4} 28.
Bf3 {+6.98/9 7} G@a7 {-1.28/9 29} 29. g5 {+5.65/9 9} Gxa8 {+1.28/8 3} 30.
Bxa8+ {+3.46/8 6} P@g3 {+0.92/7 2.9} 31. Rxg3 {+6.94/8 5} P@g4 {-0.52/7 3}
32. Rg2 {+8.20/9 6} Bc6 {-1.88/7 5} 33. +Bxc6 {+5.78/9 9} Gxc6 {-2.68/8 5}
34. B@f3 {+6.29/8 14} Gd6 {-1.80/7 8} 35. g6+ {+9.32/8 9} Ba5 {-4.24/7 4}
36. +Pxg7 {+15.30/7 6} g3+ {-4.72/6 1.8} 37. G@f7 {+11.90/8 14} Kd7
{-5.32/8 6} 38. Rxg3 {+10.56/7 7} Bxd2+ {-6.64/7 8} 39. Bg4 {+9.06/6 12}
Kc7 {+0.04/7 8} 40. P@c6 {+7.91/7 12} Kxc6 {+2.04/8 24} 41. Bxc8+
{+7.58/6 8} B@c4 {+399.84/7 0.9} 42. Kb2 {-159.94/3} R@c2 {+399.88/5 0.1}
43. Ka1 {-159.96/3} G@a2 {+399.92/3 0.1} 44. +Bxa2 {-159.98/2} Bxa2+#
{+399.96/1 0.1}
{Black mates} 0-1

Same happens here:

Code: Select all

[Event "Computer Chess Game"]
[Site "ONTWIKKELLAPTOP"]
[Date "2015.02.26"]
[Round "2"]
[White "Shokidoki euro"]
[Black "Sjaak II 1.0 RC5"]
[Result "1-0"]
[TimeControl "40/60"]
[Variant "euroshogi"]
[VariantMen "P:fW;N:sWffN;B:B;R:R;G:WfF;+P:WfF;+N:WfF;+B:BW;+R:RF;K:K;+p:WfF;+n:WfF;+b:BW;+r:RF"]
[FEN "1nbgkgn1/1r4b1/pppppppp/8/8/PPPPPPPP/1B4R1/1NGKGBN1[-] w 0 1"]
[SetUp "1"]

{--------------
. n b g k g n .
. r . . . . b .
p p p p p p p p
. . . . . . . .
. . . . . . . .
P P P P P P P P
. B . . . . R .
. N G K G B N .
white to play
--------------}
1. Kd2 {+0.40/12} b5 {-0.12/10 0.8} 2. Gc2 {+0.08/12 1.3} f5 {+0.12/10 1.1}
3. Nc1 {+1.20/11 0.9} c5 {+0.31/10 0.8} 4. g4 {+1.96/10 0.8} Bf6
{+0.39/9 1.1} 5. Nh1 {+1.64/10 1.1} Rf7 {+0.41/10 0.8} 6. Nd1
{+2.20/10 1.1} Bb7 {+0.71/9 1.2} 7. Ng3 {+2.04/9 0.8} Be5 {+0.89/9 1.1} 8.
Rg1 {+1.80/9 0.9} f4 {+1.84/10 1.7} 9. fxf4 {+1.68/9 0.6} Rxf4
{+2.00/9 1.7} 10. Bg2 {+1.36/9 2.8} Bxg2+ {+1.85/9 1.7} 11. Rxg2
{+1.32/8 0.9} B@d5 {+1.76/8 1.7} 12. P@f3 {+2.44/8 0.7} Rxg4 {+1.40/8 0.9}
13. Gf2 {+0.68/10 2.0} P@f4 {+1.65/8 1.2} 14. B@f5 {+2.48/7 0.6} Rg5
{+1.65/8 1.2} 15. Be4 {+1.08/9 0.7} Nf6 {+1.63/8 1.8} 16. Bxd5
{+1.76/8 0.2} dxd5 {+1.78/8 1.5} 17. fxf4 {+1.04/7 1.1} Ng4 {+3.52/8 1.9}
18. Gf3 {+1.96/8 0.7} Nxf4 {+3.77/7 1.2} 19. Gxf4 {+0.88/8 1.0} Bxf4
{+3.70/7 0.9} 20. P@g4 {-0.04/7 1.1} Rxg4 {+3.29/6 1.1} 21. N@f6
{-0.80/8 1.3} Kd7 {+4.20/7 2.1} 22. Nxe6+ {-1.48/8 1.4} Kc6 {+4.22/7 1.8}
23. B@e5 {-2.32/8 2.0} Bxe5 {+4.00/7 1.9} 24. +Nxe5 {-2.80/7 0.2} G@f3
{+3.77/7 1.7} 25. Rg1 {-4.84/7 1.2} B@f2 {+6.34/7 1.8} 26. B@a7
{-6.84/7 1.9} Bxg1+ {+14.30/7 1.8} 27. Nxf3 {-10.72/7 0.9} Rg2+
{+14.52/7 3} 28. Kc1 {-11.00/8 1.1} R@f2 {+15.13/7 1.8} 29. G@d2
{-11.56/8 0.8} Kb7 {+16.88/8 1.7} 30. Bxb8+ {-13.44/8 2.4} Kxb8
{+17.26/7 1.2} 31. N@c6 {-13.68/8 1.4} Ka7 {+16.96/7 1.7} 32. Nxd8+
{-13.28/8 1.3} Rxf3+ {+17.08/7 1.3} 33. +Nc8= {-13.68/7 1.4} B@g3
{+18.98/6 0.9} 34. +Nc7= {-10.76/7 1.5} B@a8 {+19.10/8 2.1} 35. G@b8
{-8.60/8 1.5} Bxe5+ {+15.46/7 1.0} 36. Gxa8= {-6.68/7 0.3} Kb6
{+10.55/8 1.1} 37. B@a7 {-6.72/8 1.3} Ka5 {+9.86/8 1.9} 38. +Nc6=
{+3.80/8 1.7} N@c8 {+10.36/7 1.8} 39. c4 {+3.84/6 0.4} +Bxb2 {+7.94/7 1.9}
40. Kxb2 {+22.60/8 2.8} N@b6 {+9.12/5 1.8} 41. B@c3 {-400.00/11 10} B@b4
{-159.90/6 1.0} 42. bxb4 {+399.80/9 0.5} Nxc4 {-159.92/6 0.1} 43. Kb3
{+399.84/7 0.2} Nb2+ {-159.94/5 0.1} 44. Kxb2 {+399.88/5 0.2} bxb4
{-159.96/2 0.1} 45. N@b3 {+399.92/3 0.1} Kb5 {-159.96/2} 46. Bxc5+#
{+399.96/1 0.1}
{White mates} 1-0

Shokidoki is also not immune for "running into the knife", though:

Code: Select all

[Event "Computer Chess Game"]
[Site "ONTWIKKELLAPTOP"]
[Date "2015.02.26"]
[Round "9"]
[White "Sjaak II 1.0 RC5"]
[Black "Shokidoki euro"]
[Result "1-0"]
[TimeControl "40/60"]
[Variant "euroshogi"]
[VariantMen "P:fW;N:sWffN;B:B;R:R;G:WfF;+P:WfF;+N:WfF;+B:BW;+R:RF;K:K;+p:WfF;+n:WfF;+b:BW;+r:RF"]
[FEN "1nbgkgn1/1r4b1/pppppppp/8/8/PPPPPPPP/1B4R1/1NGKGBN1[-] w 0 1"]
[SetUp "1"]

{--------------
. n b g k g n .
. r . . . . b .
p p p p p p p p
. . . . . . . .
. . . . . . . .
P P P P P P P P
. B . . . . R .
. N G K G B N .
white to play
--------------}
1. c4 {+0.02/10} Ke7 {+0.36/11 0.8} 2. f4 {+0.46/9 0.8} Bd7 {+1.00/10 1.3}
3. Bc3 {+0.61/9 1.6} Gf7 {+1.28/10 1.0} 4. Be2 {+0.66/10 1.7} Rc7
{+1.40/10 1.3} 5. Bf3 {+0.91/10 1.0} Nc8 {+1.68/10 1.0} 6. Kc2
{+0.96/10 1.5} c5 {+1.24/8 0.7} 7. cxc5 {+1.30/9 0.8} Rxc5 {+1.32/9 0.7} 8.
Ba8+ {+1.03/9 1.3} f5 {+2.28/7 0.7} 9. P@c4 {+0.10/7 1.4} Rxc4
{+3.52/9 1.0} 10. Ged2 {-0.88/6 1.0} fxf4 {+3.92/8 0.8} 11. Rf2
{-1.37/8 0.9} Nf6 {+3.48/8 2.5} 12. +Bb7 {-0.08/9 1.0} a5 {+3.72/8 0.8} 13.
+Ba6 {+0.44/9 1.2} b5 {+2.72/8 0.6} 14. +Bxa5 {+0.71/8 0.9} g5
{+2.44/8 1.7} 15. Kb2 {+0.73/8 1.7} g4 {+2.12/7 0.9} 16. gxg4 {+0.52/7 1.3}
P@a2 {+1.92/7 6} 17. P@f5 {+2.50/7 1.3} Nxg4 {+1.40/7 0.6} 18. Rg2
{+2.06/7 1.4} f3+ {+1.16/6 0.7} 19. Nxf3 {+3.44/7 1.8} Bc6 {+0.04/7 0.8}
20. Bxg7+ {+4.34/7 1.7} Gxg7 {-0.32/6 0.4} 21. B@e4 {+3.26/6 1.3} a1+
{-1.68/7 2.3} 22. Rxg4 {+5.56/6 1.1} +Pxb1= {-2.88/6 1.0} 23. Gxb1
{+6.10/6 3} P@g5 {-2.76/7 1.3} 24. Rxg5 {+6.35/6 1.8} P@g6 {-1.96/7 1.1}
25. Bxc6+ {+6.20/6 1.1} Rxc6 {-1.96/6 0.4} 26. +Bxb5 {+6.10/6 1.7} N@c4
{-1.56/6 0.7} 27. Kc3 {+2.97/6 1.8} B@a5 {-1.36/7 1.5} 28. +Bxa5
{+3.53/5 0.9} Nxd2+ {-3.16/7 3} 29. Kxd2 {+3.47/5 1.0} gxg5 {-0.44/5 0.6}
30. B@c3 {+2.10/4 1.4} R@h2 {+1.28/5 1.2} 31. N@e2 {+2.23/5 1.2} G@f2
{+0.72/5 0.7} 32. B@f6 {+4.00/5 1.4} Kd7 {-399.76/9 1.0} 33. N@e5
{+159.89/6 2.1} exe5 {-399.80/10 0.2} 34. Nxe5 {+159.91/6 1.0} Ke8
{-399.84/8 0.2} 35. +Bxd8 {+159.93/5 0.1} Kf8 {-399.88/6 0.2} 36. G@e7
{+159.95/4 0.1} Kg8 {+0.00/1} 37. Nf7+ {+159.97/2 0.1} Gxf7 {-399.96/2 0.2}
38. Bh8+# {+159.99/2 0.1}
{White mates} 1-0

[myfish]

Very nice! I configured Euro-Shogi with mandatory promotion now, which was the only imperfection left. RC5 also seems significantly stronger than earlier versions. Shokidoki now really has to work hard to beat it (and does not always manage that), in Euro-Shogi.

I encountered no problems so far. In the following game it is suddenly checkmated while both engines agreed it was significantly ahead, but I guess that is just lack of depth.

I've ran it without issue too (so far).

This is over many variants that play, but as you suggest, lack depth especially in the end game. I've seen sjaakii and shokidoki throw away a dragon or horse for no reason and been thinking, 'surely, the great decoyed move is coming', but...

It doesn't.

This once more brings me to my 'shogi specific' quest for parity in things like end game tables and the ability for engines to compile 'books' for perhaps (each variant). These could be specified in the 'variants.txt'.

Like

OpeningBook = /path/to/file
EndGameBook = /path/to/file

This would mean I could, in theory, get books for gorogoro, goro + knight/lance handicap, Euro and more recently the frustrating Yari variant I have working under sjaakii.

Btw: What were the 'new values for shogi pieces' you used. I was considering using 'Tanigawa's values

Pawn =1
Tokin =12

etc as seen here http://81squareuniverse.com/shogi/how-t ... ce-values/

[Evert]

I've ran it without issue too (so far).

This is over many variants that play, but as you suggest, lack depth especially in the end game. I've seen sjaakii and shokidoki throw away a dragon or horse for no reason and been thinking, 'surely, the great decoyed move is coming', but...

It doesn't.

Well, any suggestions for how to improve end-game play in Shogi are welcome. I don't even know how to define "endgame" there.
In the evaluation, SjaakII calculates game phase in the "standard" orthochess way, using a weighted sum of available material, which is then used to interpolate between "middle game" and "end game" evaluations. This is probably completely broken for games where material doesn't actually disappear.

This once more brings me to my 'shogi specific' quest for parity in things like end game tables and the ability for engines to compile 'books' for perhaps (each variant). These could be specified in the 'variants.txt'.

Like

OpeningBook = /path/to/file
EndGameBook = /path/to/file

This would mean I could, in theory, get books for gorogoro, goro + knight/lance handicap, Euro and more recently the frustrating Yari variant I have working under sjaakii.

Well, I've been thinking of adding support for variant polyglot books, but I have no idea what to actually put in them. It's a big addition though, so I'm saving it for post 1.0.

I haven't worked out all the details of how I would want this to work, so any feedback/suggestions/brainstorming here is welcome.

By the way: if you would like the various variants you have defined to be included in the release package, feel free to send me the definitions and I'll be happy to include them.

Btw: What were the 'new values for shogi pieces' you used. I was considering using 'Tanigawa's values

Pawn =1
Tokin =12

etc as seen here http://81squareuniverse.com/shogi/how-t ... ce-values/

They're defined in include/variants.h (in create_shogi_game):

Code: Select all

   float m_scale = 0.3;
   game->add_piece_type( fp,  fp, pf, pz, "+",    "Pawn",            "P,p", "P",  80*m_scale);
   game->add_piece_type( fl,  fl, 0,  pz, "+",    "Lance",           "L,l", "L", 225*m_scale);
   game->add_piece_type( fn,  fn, 0,  pz, "+",    "Knight",          "N,n", "N", 250*m_scale);
   game->add_piece_type( fs,  fs, 0,  pz, "+",    "Silver general",  "S,s", "S", 375*m_scale);
   game->add_piece_type( fb,  fb, 0,  pz, "+",    "Bishop",          "B,b", "B", 575*m_scale);
   game->add_piece_type( fr,  fr, 0,  pz, "+",    "Rook",            "R,r", "R", 650*m_scale);
   game->add_piece_type( fg,  fg, 0,  pp, "",     "Gold general",    "G,g", "G", 450*m_scale);
   game->add_piece_type( fk,  fk, kf, pp, "",     "King",            "K,k", "K");

   game->add_piece_type( fg,  fg, 0,  pp, "",     "Promoted pawn",   "+P,+p", "+P", 530*m_scale);
   game->add_piece_type( fg,  fg, 0,  pp, "",     "Promoted lance",  "+L,+l", "+L", 480*m_scale);
   game->add_piece_type( fg,  fg, 0,  pp, "",     "Promoted knight", "+N,+n", "+N", 500*m_scale);
   game->add_piece_type( fg,  fg, 0,  pp, "",     "Promoted silver", "+S,+s", "+S", 490*m_scale);
   game->add_piece_type(fdh, fdh, 0,  pp, "",     "Dragon horse",    "+B,+b", "+B", 825*m_scale);
   game->add_piece_type(fdk, fdk, 0,  pp, "",     "Dragon king",     "+R,+r", "+R", 950*m_scale);

As you can see, they have a much smaller ratio of Tokin/Pawn value (6.6). The rescaling that was done was simply including the "m_scale" factor; m_scale=1 corresponds to the old values. They can be changed in the source, but then you need to recompile to use them.

An alternative is to recreate the definition of Shogi in the config file, call it say tweaked_shogi, and then enter "shogi=tweaked_shogi" in the "variant alias" in the engine options dialog. That will cause SjaakII to select tweaked_shogi when you select variant shogi.

[myfish]

All sounds good that you are heading in the mentioned direction.

I'm not a programmer, just a player that tinkers. However, I will test stuff relentlessly and report feedback.

As for these variations, sure. I have no problem sharing them and of course, as I tweak them, I'm sure you'd update them.

I even have a bunch of SVG pieces for shogi and a few variants like Chu, Tori and Yari.

Of course, these won't really help people outside the xboard interface but you are welcome to use them.

I'm playing with the dobutsu config now you've added the pawn dead drop and I'll make some SVG pieces for that too in the spirit of the original game.

What I'm actually after regards books etc is the ability to define a variant to use, play certain openings so I can practice against certain opening lines.

HGM has already tried to make sense of my drivel in places as to what would and wouldn't work. We've got into all sorts of issues regards HOW a book would be made.

Xboard spouts pgn and reads the moves in psn. However, unlike chess where there are a million tools, shogi is psn, pgn,kif, ki2 and even then, some are Shift_JIS encoded and others UTF-8.

If I could collect, games opening in say 'nakabisha', I could get a 1000 nakabisha kif's and create a 'nakabisha' opening book. It would take work, I do not doubt this at all.

However, IF the mechanism was in place, I think it would be useful. Just like scid lets you choose or force openings.

End games yes, a nightmare as material is thinned and thinned usually in chess before the end game and end game specifics. This is not forced to happen unless you withhold material in hand in shogi.

But, there are tens of thousands of tsumes of various depths and threatmate, brinkmating methods that I believe could be used.

Even tesuji or (good moves). Surely, an engine could scan a tesuji book for 'middlegame' where it relate book moves to the positions on the board being played. It could then decide whether it was in it's interest to follow the procedures outlined in that book to gain position or material advantage.

I'm probably not explaining it in a way a programmer would actually do it, sorry.

[hgm]

Well, any suggestions for how to improve end-game play in Shogi are welcome. I don't even know how to define "endgame" there.
In the evaluation, SjaakII calculates game phase in the "standard" orthochess way, using a weighted sum of available material, which is then used to interpolate between "middle game" and "end game" evaluations. This is probably completely broken for games where material doesn't actually disappear.

Indeed, Shogi end-games are not end-games in the Chess sense, but more a 'race to mate first'. I think special threat extensions and recognition of spite checks and futile interpositions can improve game play here.

Perhaps the following game can illustrate where general improvements are possible. Sjaak is doing very well, building up a huge lead against Shokidoki. Sjaak even seems to outsearch Shokidoki, and often reaches 8 nominal plies where Shokidoki only reaches 7.

And then in a single move (42... P@d2) it basically throws away the game. Shokidoki counters with 43. B@c7, enclosing the black King (and attacking the Dragon as a side dish). It cannot be stressed enough how dangerous it is to have enemies on both sides of the King; in Shokidoki I really give a huge extra penalty for that in the King Safety. The same material distributed over two sides is much more dangerous as when it only comes from one side. This is a general property of drop games, where any presence acts as a 'nucleation site' for a full-scale drop invasion.

Code: Select all

[Event "Computer Chess Game"]
[Site "ONTWIKKELLAPTOP"]
[Date "2015.02.26"]
[Round "4"]
[White "Shokidoki euro"]
[Black "Sjaak II 1.0 RC5"]
[Result "1-0"]
[TimeControl "40/1200"]
[Variant "euroshogi"]
[VariantMen "P:fW;N:sWffN;B:B;R:R;G:WfF;+P:WfF;+N:WfF;+B:BW;+R:RF;K:K;+p:WfF;+n:WfF;+b:BW;+r:RF"]
[FEN "1nbgkgn1/1r4b1/pppppppp/8/8/PPPPPPPP/1B4R1/1NGKGBN1[-] w 0 1"]
[SetUp "1"]

{--------------
. n b g k g n .
. r . . . . b .
p p p p p p p p
. . . . . . . .
. . . . . . . .
P P P P P P P P
. B . . . . R .
. N G K G B N .
white to play
--------------}
1. g4 {+0.32/15} b5 {-0.11/17 23} 2. Gc2 {+0.32/15 16} Gf7 {-0.09/16 23} 3.
Be2 {+0.08/15 16} Gde7 {+0.10/15 24} 4. Kd2 {+1.36/15 19} f5 {+0.23/16 24}
5. g5 {+1.48/15 24} gxg5 {+0.26/16 31} 6. Rxg5 {+0.88/14 47} e5
{+0.27/14 31} 7. Nf1 {+1.16/12 19} P@g6 {+0.99/14 31} 8. Rg1 {+1.32/14 19}
b4 {+1.11/16 21} 9. bxb4 {+0.84/14 17} Be6 {+1.04/16 31} 10. Nc1
{+0.48/13 35} Ba2+ {+1.16/15 27} 11. P@g5 {+0.72/12 16} gxg5 {+1.79/13 16}
12. Rxg5 {+0.16/13 23} +Be6 {+1.40/13 23} 13. d4 {+1.12/12 27} Rxb4
{+2.49/12 30} 14. Bc4 {+1.20/11 17} +Bf6 {+2.36/12 45} 15. Rg1
{+0.96/10 16} e4 {+2.53/11 31} 16. P@g6 {+1.00/10 21} Bh8 {+3.15/10 18} 17.
P@b3 {+0.24/10 31} Rb7 {+3.50/12 32} 18. Ge2 {+0.00/10 17} +Bxd4
{+3.33/11 20} 19. Ged3 {-0.48/10 44} exe3+ {+4.33/11 38} 20. Gxe3
{-0.80/11 43} +Bb6 {+4.43/11 27} 21. P@e6 {+0.72/9 14} Gef6 {+5.47/11 33}
22. Rg2 {-0.32/9 15} Nf8 {+6.17/11 28} 23. b4 {-1.12/9 30} Nxe6
{+6.15/11 33} 24. b5 {-1.36/9 20} +Ba7 {+6.22/11 33} 25. Bb3 {-1.32/9 45}
Rxb5 {+7.43/11 20} 26. c4 {-1.80/9 20} d5 {+7.58/10 21} 27. Nd1
{-2.20/9 42} Nd4 {+8.27/10 18} 28. Bxd4 {-2.80/9 24} dxd4 {+10.30/10 26}
29. N@g8 {-2.76/8 12} P@e4 {+8.64/8 23} 30. Ge2 {-3.12/9 24} d3+
{+9.55/8 25} 31. Gexd3 {-3.12/8 8} B@h1 {+8.05/8 39} 32. Rg3 {-4.92/10 16}
P@g4 {+12.11/10 45} 33. Rxg4 {-6.60/10 35} P@d4 {+13.24/10 1:05} 34. Gdc3
{-7.76/9 52} Bxf3+ {+12.76/9 20} 35. Rg3 {-10.08/9 51} +Bxg3 {+16.05/9 34}
36. Nxg3 {-9.00/9 55} R@g2 {+16.05/8 25} 37. Kc1 {-10.36/10 1:07} d3+
{+16.65/9 38} 38. G2xd3 {-7.60/9 24} Rxb3+ {+17.70/9 40} 39. Gxb3
{-7.84/8 16} Rxg3+ {+17.00/8 28} 40. Gbc3 {-8.40/8 19} N@b3 {+18.50/8 51}
41. Kb2 {-9.08/8 33} Nxc3+ {+18.02/8 1:06} 42. Gxc3 {-9.52/8 31} P@d2
{+19.15/7 17} 43. B@c7 {-4.96/8 37} P@b3 {+15.96/7 18} 44. Kxb3
{-2.96/8 18} P@b4 {+15.86/8 30} 45. Kb2 {-3.56/8 15} b3+ {+13.87/8 30} 46.
Kxb3 {-0.88/8 15} P@b4 {+10.26/8 30} 47. Kb2 {-1.96/8 38} G@b3 {+8.92/8 30}
48. Kb1 {-1.16/8 32} B@e7 {+6.71/7 28} 49. Bxg3+ {+1.88/8 13} dxd1+
{+2.56/9 33} 50. +Bc7= {+3.04/8 27} Gf8 {+1.87/8 29} 51. R@h7 {+20.76/8 45}
+Pc1 {-0.27/8 30} 52. Kxc1 {+22.32/8 19} +Be3 {-5.78/8 30} 53. Kb1
{+31.40/8 33} Gb2 {-15.36/8 30} 54. Kxb2 {+0.00/1 0.1} b3+ {-27.63/8 30}
55. Kxb3 {+33.20/7 8} N@a5 {-37.67/8 30} 56. Ka4 {+55.08/8 1:02} Bxa3+
{-54.59/8 20} 57. Kxa3 {+0.00/1 0.1} +Bc1 {-159.88/9 25} 58. Ka2
{+399.72/8 6} +Bb1 {-159.88/8 16} 59. Kxb1 {+399.80/9 0.4} P@b2
{-159.90/7 0.2} 60. Kc1 {+399.84/7} b1+ {-159.92/6 0.1} 61. Kxb1
{+399.84/7 0.2} Bg7 {-159.94/5 0.1} 62. Nxf8+ {+399.88/5 0.1} Kxf8
{-159.96/2 0.1} 63. Rxg7+ {+399.92/3 0.1} Ke8 {-159.98/2 0.1} 64. G@f8#
{+399.96/1 0.1}
{White mates} 1-0

A possible search improvement is the recognition of 'pseudo-repeats', which occur in this game as 44... P@b4 45. Kb2 Pb3+ 46. Kxb3. This reaches the same board position, but transferred a Pawn from the hand of Sjaak to that of Shokidoki. It delays the inevitable 4 ply (2 in depth after check extension) at the cost of a Pawn, and as Sjaak has 2 Pawns in hand it greatly helps to make it blind for how dangerous 43. B@c7 actually was. The threat here was 44. R@d8 {contact check} Ke7 {only move} 45. Rd6+ {contact check} Ke8 {only move} 46 +Rd8#, and it seems there is no defense against the mate, but the spite checks with the Pawn make that invisible. This could be avoided by hard pruning of moves that cause a pseudo-repeat with a subset of the original hand left in the hand.

[Evert]

All sounds good that you are heading in the mentioned direction.

I'm not a programmer, just a player that tinkers. However, I will test stuff relentlessly and report feedback.

That's fine. If you know how to build the code, you can easily tweak the piece values by changing the numbers, if you wanted to.

I even have a bunch of SVG pieces for shogi and a few variants like Chu, Tori and Yari.

Of course, these won't really help people outside the xboard interface but you are welcome to use them.

Well, I guess Tori pieces for XBoard would be quite welcome.

What I'm actually after regards books etc is the ability to define a variant to use, play certain openings so I can practice against certain opening lines.

HGM has already tried to make sense of my drivel in places as to what would and wouldn't work. We've got into all sorts of issues regards HOW a book would be made.

Well, there are two issues here: from my perspective, the challenge is to make it possible for Sjaak to use an opening book for any given variant.
This is a challenge, because Sjaak doesn't "know" anything about the games it plays. It doesn't know that a knight is a knight, it just knows that there is a piece that makes a (1,2) leap. For the polyglot books I mentioned (which I think is what XBoard can make) this presents a difficulty, because you do need to know how to map piece types to the book format.

I guess the most basic and portable book format is to have a text file with one position per line, described by a FEN position string, followed by a list of possible moves in that position (perhaps including a frequency indication for how often a move should be selected). I worry about the speed of processing such a book though, but perhaps that's something to try.

If I could collect, games opening in say 'nakabisha', I could get a 1000 nakabisha kif's and create a 'nakabisha' opening book. It would take work, I do not doubt this at all.

However, IF the mechanism was in place, I think it would be useful. Just like scid lets you choose or force openings.

For sure. SjaakII allows you to define any variant you like, so clearly it must not rely on opening books to play a semi-decent opening, but I'm not going to claim that it does this particularly well.
Either way, it clearly needs some way to benefit from opening books, if they're available.

But, there are tens of thousands of tsumes of various depths and threatmate, brinkmating methods that I believe could be used.

SjaakII now has a tsume solver (it's the mate-solver I mentioned), but it is restricted in when it's called and how deep a mate it looks for to keep the cost down. No doubt I could improve it.

If you give it a tsume test position and let it analyse, you should be able to see a difference between having the mate-solver enabled or not (there's an option for that in its settings dialogue).

Even tesuji or (good moves). Surely, an engine could scan a tesuji book for 'middlegame' where it relate book moves to the positions on the board being played. It could then decide whether it was in it's interest to follow the procedures outlined in that book to gain position or material advantage.

I'm probably not explaining it in a way a programmer would actually do it, sorry.

That's ok. Some programmers have a tendency to outline ideas only in pseudo code, which isn't always helpful.

The difficulty with matching moves and strategies in "similar" positions is how to define "similar". For humans it's sortof obvious, for computers, not so much. So you will need to describe the position, identify the defining characteristic and indicate fairly precisely what the next step in the plan should be.
If you have links that could help for this that would be great (I find it extremely hard to find relevant Shogi links that have enough detail for me to follow, but don't assume too much familiarity with the game).

[Evert]

Well, any suggestions for how to improve end-game play in Shogi are welcome. I don't even know how to define "endgame" there.
In the evaluation, SjaakII calculates game phase in the "standard" orthochess way, using a weighted sum of available material, which is then used to interpolate between "middle game" and "end game" evaluations. This is probably completely broken for games where material doesn't actually disappear.

Indeed, Shogi end-games are not end-games in the Chess sense, but more a 'race to mate first'. I think special threat extensions and recognition of spite checks and futile interpositions can improve game play here.

Mate-threat extensions should be easy to add (null move will identify them), recognising spite checks and useless interpositions is not so easy (as I'm sure you don't need me to tell you). I guess I could make creative use of the mate-solver to explicitly check whether my last move was a spite check (mate-threat detected two ply earlier, my move in that node was a checking move that was answered by an immediate capture -> call mate search to test if the mate threat still exists; if so speculatively prune the current branch assuming the evasion was futile unless the remaining depth is very large, just so I don't throw away perfectly good evasions that may be possible now).

It'll require some work though (and preferably a library of good test positions).

Out of general interest: do you define "phase" in Shokidoki's eval at all? I'm wondering if I shouldn't just stick with the "mg" eval in drop games (promotions can make the material phase counter larger than it was at the beginning of the game, causing the evaluation to extrapolate the evaluation terms to "before" the beginning of the game, which probably unbalances things). Something to try, I guess.

Perhaps the following game can illustrate where general improvements are possible. Sjaak is doing very well, building up a huge lead against Shokidoki. Sjaak even seems to outsearch Shokidoki, and often reaches 8 nominal plies where Shokidoki only reaches 7.

And then in a single move (42... P@d2) it basically throws away the game. Shokidoki counters with 43. B@c7, enclosing the black King (and attacking the Dragon as a side dish). It cannot be stressed enough how dangerous it is to have enemies on both sides of the King; in Shokidoki I really give a huge extra penalty for that in the King Safety. The same material distributed over two sides is much more dangerous as when it only comes from one side. This is a general property of drop games, where any presence acts as a 'nucleation site' for a full-scale drop invasion.

Ah, that is exactly the sort of pointer I need (I particularly like general eval terms that are more general than orthchess or shogi, although I suppose I effectively have Xiangqi-specific terms in there as well)! I'll take a close look at the game as well.

A possible search improvement is the recognition of 'pseudo-repeats', which occur in this game as 44... P@b4 45. Kb2 Pb3+ 46. Kxb3. This reaches the same board position, but transferred a Pawn from the hand of Sjaak to that of Shokidoki.

That would be good to detect, but it's a little bit tricky: I would need to keep track of the "board key" and "holdings key" independently (not a big issue) and then be able to query the difference in holdings between the two moves (that I can't easily do at the moment, I would need to unwind part of the move stack).
I suppose I would just fail low hard if such a situation occurs. Another useful thing to do in that situation (if I don't want to fail low) would be to graft the killers from the earlier position onto the current ply.

The threat here was 44. R@d8 {contact check} Ke7 {only move} 45. Rd6+ {contact check} Ke8 {only move} 46 +Rd8#, and it seems there is no defense against the mate, but the spite checks with the Pawn make that invisible. This could be avoided by hard pruning of moves that cause a pseudo-repeat with a subset of the original hand left in the hand.

In this particular case, the mate-search wouldn't have helped either: even if it would have triggered (it needs 8 ply of remaining depth to trigger), it is limited to 3-ply (except near the root, where it starts at 3 ply and grows by 2 ply every 4 iterations, the goal is mainly to prevent SjaakII from overlooking mates that could easily be avoided). Pruning futile interpositions would be useful though. I'll look into that.

[hgm]

Out of general interest: do you define "phase" in Shokidoki's eval at all? I'm wondering if I shouldn't just stick with the "mg" eval in drop games.

I don't distinguish game phase in Shokidoki.

Ah, that is exactly the sort of pointer I need (I particularly like general eval terms that are more general than orthchess or shogi, although I suppose I effectively have Xiangqi-specific terms in there as well)! I'll take a close look at the game as well.

To play Xiangqi well you need very specific eval terms, which would be difficult to detect by a heuristic, because they are the consequence of a combination of factors. E.g. Cannons are very vulnerable to Rooks, because they have the same sliding motion. So it is difficult to escape an attack. A Cannon soft-pinned between opposing Rooks along a file can be an unsolvable problem on some files, because it can be impossible to protect the Rook behind it: It could be outside the Palace, and the Elephants might skip over the file, so that there are only few pieces that could in principle protect it so you can move the Cannon away. Also attacking is easier than defending, so the entire game is a race to mate. But the presence of purely defending pieces might make an attack futile, and it is important to recognize when this is the case. E.g. if you have one Horse, and the opponent has two Horses + two Advisers, attacking is futile, as H vs 2A is already draw (IIRC). 2H vs an unshielded K is an easy win, though, so you'd better keep your horse back for defense, where it can do some good. That only works because the A are confined, and cannot join the attack if you stop harassing the King. Basically you should make your evaluation per board half, where you top off an advantage for the defender to 0, and then sum the board halves.

An eval term that could be of general use is associated with 'lame' pieces like H and E: they are very easily trapped by blocking them. So safe mobility of these should have a quite large weight.

That would be good to detect, but it's a little bit tricky: I would need to keep track of the "board key" and "holdings key" independently (not a big issue) and then be able to query the difference in holdings between the two moves (that I can't easily do at the moment, I would need to unwind part of the move stack).
I suppose I would just fail low hard if such a situation occurs. Another useful thing to do in that situation (if I don't want to fail low) would be to graft the killers from the earlier position onto the current ply.

Indeed, it is important to have separate board and holdings key for that reason. In Shokidoki I use a hack: rather than storing the full holdings state in the table of repeat keys, I store the eval. If the eval differs by exactly two Pawns in hand I must be burning Pawns, and otherwise I must be burning pieces. If I lose twice the value of the weakest piece or more, it is very unlikely my material is changing, rather than just depleting. I think failing low (even against a mated window) is the best option. It is really a 'never do' thing, you cannot possibly benefit from transferring material to his hand. So you might as well consider it an illegal move. Of course trading R in hand for N in hand might change a lost position in a win.

In this particular case, the mate-search wouldn't have helped either:...

True, it is a very hard problem. But the appearence of B@c7 while Ng8 blocks any escape to that side is very bad business, and more often than not lethal. This is what makes Shogi so difficult. Even when your King is doomed between B and N, you still would not care when you can mate the other side first. You can only know that by calculating very precisely, and when the mates are outside the horizon there really is no way to know. You just know that one way or the other the game is decided. This is why it is so important to try to get the mate inside the horizon by pruning the moves that are just pointless delays.