Nebiyu hash (Daniel)

[hgm]

I am running a tourney with 64MB hash (hash + EGTB cache = 68MB). But in the task manager I see that NebiyuAlien has a 111 MB memory footprint. It turns out that NebiyuAlien does not respond to the WB memory command, and is honest enough about that, by sending 'feature memory=0'.

OK, this is annoying but compliant, and the user would have to go through the cumbersome chore of editing ini files to set the hash size. But in the Nebiyu.ini file the hash is already set to 64MB. NebiyuAlien seems to ignore that ini file!

So how can the hash size of NebiyuAlien be set? Would it be a terrible problem to make it respond to the memory command?

[Daniel Shawul]

I am running a tourney with 64MB hash (hash + EGTB cache = 68MB). But in the task manager I see that NebiyuAlien has a 111 MB memory footprint. It turns out that NebiyuAlien does not respond to the WB memory command, and is honest enough about that, by sending 'feature memory=0'.

It looks like I am allocating the right amount of space but the usage rises after first search. Maybe there is some other table I forgot about. Scorpio starts from small footprint of 1mb and then rises to 64mb after first search, but nebiyu starts from 64mb and adds another 64 so it maybe a memory leak. Anyway I will fix that one. Supporting memory is much easier for nebiyu because the only table is the main hash table. No pawn or egtb caches, so I just added recognition of 'memory' besides 'ht' and seems to work. I also enabled 'smp=1' to let winboard control it. It will read the ini file first, so winboard should send them after the engine is started I guess...

OK, this is annoying but compliant, and the user would have to go through the cumbersome chore of editing ini files to set the hash size. But in the Nebiyu.ini file the hash is already set to 64MB. NebiyuAlien seems to ignore that ini file!

So how can the hash size of NebiyuAlien be set? Would it be a terrible problem to make it respond to the memory command?

[Daniel Shawul]

Surprisingly the problem is not related to the hash tables, but just that the Checkers and Alien version have a very large MOVE data structure that makes those exes extremely large when loaded in RAM, 18mb and 53mb resp. The other versions, which use 4 bytes per move, all consume less than 1mb when loaded. I still don't know why it is so big though since the checker's MOVE struct consume only about 4x, but exe is much larger. Maybe there are some struct alignment issues ...

[Daniel Shawul]

I have also fixed the 'white looses on three-fold repetition rule' of minishogi now and it seems to understand the game you posted before in which Nebiyu was happy to repeat. There is nothing much I can do about the sizes of the exes right now because it seems there are data structures other than MOVE that makes checkers/alien versions a bit 'heavy'. But I still don't clearly understand how that added up to 52 mb ...

Notes:
-mini-Shogi package expects Nebiyu to define the game as 5x5+5_shogi.
-The bigchess engine list contains _all_ engines including those that don't support it. The other variants mini-shogi and judkins have a filtered engine list.

[hgm]

Well, if it adds a fixed 40MB it is not a big deal. By modern standards that is not much. As long as the user does have a way to change the hash size.

I guess you use this move struct in the move lists, and also reserve a very large number of moves per ply level on that. 40 MB / 20 ply would be 2MB per ply, but you could reach that with 1024 moves / ply, and 512 possible 4-byte legs per move. But I don't know if NebiyuAlien works that way.

If I would have to program a similar task I would put the move lists on a separate stack, rather than in the stack frame for the node, so that they only consume memory for moves that are actually possible in all the active nodes, and somehow try to economize on the space that has to be reserved for the legs, so that in cases were all moves have just a few legs, the moves would be stored more compactly. (e.g. in stead of a fixed-size array of legs as part of the move struct take a pointer to a separate stack where all the legs are stored.)

As to the mini-Shogi package: Indeed, it contains WinBoard 4.7.2, which does not have the engine-define-variant-names patch yet. None of the other engines included would understand 'variant minishogi' anyway. But that is no problem; the engine-defined name is defined in the alien.ini file, and I simply set it to 5x5+5_shogi (with parent shogi) there. That is the advantage of not automatically adding any prefixes; it is more flexible, because the user can now decide where he wants to add prefixes, and where not. (Downside is of course that he can do it inconsistently.)

I will have a look at the Big Chess configuring. (Which doesn't really belong in this package, but as it doesn't involve any other engines, I figured I might as well piggy-back it on this one. The logical thing to do would be to make an upgrade of the WinBoard-Nebiyu package I host with the Alien Edition, which I have not upgraded for a very long time, and include Big Chess there. I first want to patch the Alien Edition for the engine-defined names as well, however.

[hgm]

Something else. I did a 5+1 tournament with 4 mini-Shogi engines (Nebiyu, Shokidoki, TJshogi5x5 and Petit Shogi) with 168 games per pairing. Nebiyu is a quite stable engine, but yet I did observe 3 crashes in its 672 games. I will post the games here, not sure if they are useful to you. (This was the version before you fixed the repetitions. The games were started from an opening line, but it is not easy to see where, as the opening lines also contained score/depth info.)

Code: Select all

[Event "Computer Chess Game"]
[Site "MAKRO-PC"]
[Date "2013.09.09"]
[Round "35"]
[White "NebiyuAlien_1.45"]
[Black "ssp (USI2WB)"]
[Result "0-1"]
[TimeControl "300+1"]
[Variant "shogi"]
[FEN "rbsgk/4p/5/P4/KGSBR[-] w 0 1"]
[SetUp "1"]

{--------------
r b s g k
. . . . p
. . . . .
P . . . .
K G S B R
white to play
--------------}
1. Sd2 Sb4 {+0.20/16 5:41} 2. Bc2 {-0.28/17 3:44} Bc4 {+0.28/17 1:48} 3.
Gb2 {-0.32/17 3:44} Gd4 {+0.32/18 19:02} 4. Rd1 {-0.32/16 3:55} Be2
{+0.28/17 4:01} 5. Rc1 {-0.36/18 2:12} Bc4 {+0.92/18 21:49} 6. Re1
{-0.68/17 3:08} Bd5 {+0.68/17 3:53} 7. Rd1 {-0.80/17 4:28} Bc4
{+1.24/16 3:47} 8. Bb1 {-0.96/16 8:00} Be2 {+1.24/16 3:22} 9. Re1
{-0.36/17 4:01} Bc4 {+0.00/17 21:37} 10. Sc1 {+0.00/16 3:00} Sc5
{+0.16/16 2:43} 11. Rd1 {-0.44/16 28:05} Rb5 {+0.44/16 2:20} 12. Bc2
{-0.12/16 4:13} Be2 {+0.00/16 4:05} 13. Re1 {+0.00/16 6:23} Bd3
{-0.04/16 3:01} 14. Bxd3 {+0.24/16 3:43} Gxd3 {+0.00/16 4:37} 15. Rd1
{-0.04/15 6:09} B@e2 16. Re1 {-0.34/21 10} e3 {+4.15/13 15} 17. Sc2
{+0.52/19 13} Rxb2 {+3.00/13 14} 18. Kxb2 {+0.18/18 9} Gxc2 {-1.40/13 15}
19. Kxc2 {+1.18/16 7} Sb4 {-5.25/12 13} 20. B@a1 {+0.12/16 8} G@c3
{-8.80/14 13} 21. Bxc3 {+1.18/20 8} Sxc3 {-0.50/14 12} 22. Kxc3
{+1.00/19 8} B@a5 {+3.80/13 12} 23. S@b4 {+0.02/20 8} S@c4 {-11.40/14 11}
24. Kc2 {+0.56/20 7} Bxb4 {+0.60/13 11} 25. Rxe2 {-1.32/19 11} S@c3
{-1.45/13 10} 26. Kd1 {-0.86/20 8} exe2 {+7.55/13 10} 27. R@e3 {-1.18/19 8}
Kd4 {+14.35/14 11} 28. G@e4 {-1.40/18 5} Kc5 29. G@d5 {-20.68/18 10} Kb5
{+72.90/14 9} 30. Gc5 {-299.83/19 5} Bxc5 {+82.35/14 9} 31. Rxe2
{-299.85/20 6} Sd3 {+88.70/13 8}
{Error: first chess program (NebiyuAlien) exited unexpectedly} 0-1

Code: Select all

[Event "Computer Chess Game"]
[Site "MAKRO-PC"]
[Date "2013.09.10"]
[Round "183"]
[White "NebiyuAlien_1.45"]
[Black "Shokidoki Yokohama 2013"]
[Result "0-1"]
[TimeControl "300+1"]
[Variant "shogi"]
[FEN "rbsgk/4p/5/P4/KGSBR[-] w 0 1"]
[SetUp "1"]

{--------------
r b s g k
. . . . p
. . . . .
P . . . .
K G S B R
white to play
--------------}
1. Sc2 Gd4 2. Gb2 Bd3 3. Sxd3 {+1.64/16 4:38} Gxd3 {-1.96/16 3:06} 4. Be2
{+1.48/16 27:55} Gxe2 {-1.44/15 3:54} 5. Rxe2 {+1.64/14 2:55} S@d4
{-1.72/14 3:46} 6. B@b1 {+1.24/14 4:28} B@d5 {-1.00/15 3:35} 7. Re1
{+0.80/16 7:30} Rb5 {-0.84/16 3:56} 8. G@c2 {+1.16/16 3:40} Bc4
{-1.20/16 4:07} 9. Ga3 {+1.00/16 2:46} Ra5 {-1.20/16 2:20} 10. Gb2
{+1.16/17 15:19} Bd5 {-0.68/17 4:00} 11. Rd1 {+0.52/17 4:19} Rb5 12. Gc2
{+0.94/19 7} e3 {-0.60/13 10} 13. Gb2 {+1.26/21 7} Bc4 {-0.40/14 11} 14.
Re1 {+1.38/20 7} Bd5 {+0.24/13 14} 15. Rc1 {+0.12/21 7} e2 {-0.72/13 11}
16. Ga4 {+0.22/22 13} Rxb2 {-0.64/14 14} 17. Kxb2 {+1.10/21 6} G@d2
{-0.72/13 9} 18. R@e3 {+1.82/22 17} Sxe3 {-0.64/13 1.5} 19. Rxc5+
{-0.44/22 9} R@b3 {-400.00/13 1:02} 20. Ka1 {-0.74/21 8} Rxb1+
{+0.72/13 36} 21. Kxb1 {-0.34/20 12} Sd4 {+0.72/11 4} 22. +Rxd5
{-0.46/18 5} Kxd5 {+0.00/1 0.1} 23. R@b5 {-2.90/17 8} Ke4 {+3.16/10 5} 24.
B@d5 {-4.72/18 5} Ke3 {+3.68/9 1.0} 25. Kb2 {-5.22/19 9} Gd3 {+3.88/9 3}
26. Bc4+ {-3.80/16 6} R@c2 {+7.00/9 5} 27. Kb3 {-6.72/19 5} B@a1
{+7.68/10 4} 28. +Bd5 {-11.38/18 6} Kd2 {+7.80/11 6} 29. Ka3 {-4.68/18 5}
e1+ {+9.08/12 7} 30. Rb1+ {-3.42/19 5} Rc1+ {+9.40/11 5} 31. +Rxc1
{-5.06/20 6} Kxc1 {+9.80/11 0.9} 32. R@b5 {-5.20/20 12} Sc3 {+10.72/10 4}
33. S@c4 {-5.68/17 4} Gc2 {+11.60/11 8} 34. +Be4 {-5.30/18 5} +Pd1
{+12.00/11 6} 35. Ga5 {-5.28/17 4} Bb2+ {+13.48/10 5} 36. Rxb2+
{-4.28/19 5} Gxb2 {+13.80/11 14} 37. B@e3 {-4.34/18 5} +Pd2 {+17.44/11 14}
{Error: second chess program (NebiyuAlien) exited unexpectedly} 0-1

Code: Select all

[Event "Computer Chess Game"]
[Site "MAKRO-PC"]
[Date "2013.09.10"]
[Round "296"]
[White "TJshogi5x5 0.19"]
[Black "NebiyuAlien_1.45"]
[Result "1-0"]
[TimeControl "300+1"]
[Variant "shogi"]
[FEN "rbsgk/4p/5/P4/KGSBR[-] w 0 1"]
[SetUp "1"]

{--------------
r b s g k
. . . . p
. . . . .
P . . . .
K G S B R
white to play
--------------}
1. Sb2 Sb4 2. Sb3 Sxb3 {+0.44/15 1:59} 3. Bxb3 {-0.20/16 5:00} Bc4
{-0.24/16 25:41} 4. Bxc4 {+0.08/15 3:43} Gxc4 {-0.28/15 5:50} 5. B@b2
{+0.28/13 2:59} S@c3 {-0.32/14 3:24} 6. Bxc3 {+0.52/15 6:28} Gxc3
{-0.52/15 3:13} 7. Rd1 {+0.36/15 9:15} B@e3 {-0.44/13 2:31} 8. S@c2
{+1.00/15 6:02} Gxc2 {-1.04/14 2:49} 9. Gxc2 {+1.04/13 24:44} S@c4
{-1.52/13 19:03} 10. G@b2 {+1.68/13 1:44} B@d4 11. S@c3 {+1.00/13 5} Rd5
{-0.06/21 8} 12. Rxd4 {+0.90/13 5} Rxd4 {+0.06/21 7} 13. Sxd4 {-0.10/13 12}
Bxd4 {+0.00/20 9} 14. R@a5 {+0.10/13 13} S@c5 {+0.00/21 9} 15. Gc1
{+1.00/12 5} R@e1 {+0.24/21 7} 16. B@d2 {+1.30/13 4} Re2+ {+1.10/22 7} 17.
Ra3+ {+0.54/14 5} +Rxd2 {+1.44/23 7} 18. Gxd2 {-1.00/13 4} B@c1
{+0.98/22 8} 19. R@c2 {-1.26/13 5} Bcxb2+ {+0.72/21 8} 20. +Rxb2
{-1.30/14 8} G@b3 {+0.94/20 7} 21. B@c1 {-1.36/13 4} Gxc2 {+0.56/21 6} 22.
Gxc2 {-0.50/13 5} R@d1 {+2.02/21 7} 23. G@c3 {-0.10/15 9} Sxc3 {+0.96/21 8}
24. Gxc3 {+0.00/14 5} G@b3 {+0.94/20 7} 25. Gxd4 {+0.00/13 5} Sxd4
{+1.88/20 6} 26. +Rxb3 {+0.00/13 4} Rxc1+ {+2.78/22 7} 27. G@b1
{+0.00/13 6} B@c3 {+2.82/23 7} 28. S@b2 {-0.20/13 4} Bxb2 {+2.50/23 5} 29.
+Rxb2 {-0.20/3 0.1} +Rxb2 {+2.62/22 6} 30. Gxb2 {-0.50/13 7} R@c1
{+2.72/20 6} 31. B@b1 {-1.00/14 10} Rc5+ {+2.94/19 5} 32. B@a3 {-1.50/13 6}
+Rd5 {+3.22/20 6} 33. Gb3 {-0.70/14 8} G@b5 {+2.80/19 5} 34. Bb2
{-0.70/13 6} S@c4 {+3.66/20 5} 35. Gxc4 {-0.70/13 5} Gxc4 {+4.82/17 4} 36.
S@c2 {-1.20/12 3} G@d2 {+5.22/19 6} 37. R@a4 {-2.90/11 4} Gxc2 {+6.50/20 5}
38. Bxc2 {-5.10/14 8} S@c3 {+6.42/19 5} 39. G@c1 {-5.10/14 10} Sxb2
{+6.44/20 7} 40. Gxb2 {-5.10/14 4} Sc3 {+7.16/20 4} 41. Gxc3 {-5.30/13 20}
Gxc3 {+7.16/18 4} 42. S@c1 {-11.10/12 18} G@d2 {+8.06/17 4} 43. Bxe4
{-6.94/11 4} +Rxe4 {+8.56/17 4} 44. Ra5+ {-6.70/10 2.0} B@d5 {+10.84/18 5}
45. Sxd2 {-6.40/11 2.5} Gxd2 {+11.22/18 4} 46. G@b2 {-7.30/11 5} S@c1
{+13.96/17 4} 47. +Rxd5 {-11.10/9 2.3} +Rxd5 {+17.96/18 3} 48. B@a3
{-11.74/10 2.1} R@d1 {+25.90/18 4} 49. P@b1 {-199.80/11 2.9} Sxb2+
{+35.30/20 3} 50. Bxb2 {-199.82/11 1.9} Ke4 {+35.42/19 3} 51. S@c1
{-199.84/11 4} Gxc1 {+51.04/16 4} 52. Bxc1 {-199.86/10 11} B@b2
{+299.86/18 4} 53. bxb2 {-199.88/10 6} G@b1 {+299.88/19 4} 54. Kxb1
{+6.16/3} Rxc1+ {+299.90/21 4} 55. Kxc1 {+9.16/3 0.1} B@e3 {+299.90/20 4}
56. Kc2 {-199.94/6}
{Error: first chess program (NebiyuAlien) exited unexpectedly} 1-0

BTW, counting the first 108 games of each pairing (after that, Windows spondtaneously started to use a non-5x5-enabled version of UCI2WB, so that all Petit-Shogi games were messed up), the score percentages were:

Code: Select all

Shokidoki Yokohama     ### 65% 81% 73%
Nebiyu 1.45            35% ### 66% 64%
TJshogi5x5 0.19        19% 34% ### 57%
Petit Shogi            27% 36% 43% ###

The Shokidoki-Nebiyu result includes 9 false draw claims (repetitions) by Nebiyu.

[Daniel Shawul]

It seems to the C++ constructor of the MOVE structure that is causing this. I set values of member variables to zero, which is the default anyway, which causes it to 'touch' the whole move stack at program start up. Without the constructor, the Checkers version shows a much smaller size of 0.5mb, even smaller than the chess version. Right now the zero-constructor seems to be needed but I am sure I will find a way to avoid it.

I was not able to reproduce the crashes but I guess it is probably because I fixed the repetition bug. After that it seems Nebiyu can barely win playing as white. Shokidoki is still strong for it but I am wondering if I haven't introduced a bug because it could not manage to win with white for 20 games now ...

Edit: It is fixed now. And the alien version now uses 1mb at startup and goes up to 3mb on first search, which is much smaller than 60mb.

[Daniel Shawul]

I have uploaded bug-fixed versions at the same place.