New Giraffe (Sept 8)

[matthewlai]

With teaches itself I think about unsupervised learning. But it is not.

Neural networks are not much fun for you start the tuner and sit and wait.
Quite boring or not ? I don't like tuning at all.

But I would like to know the limits of this neural network approach. How long do you have to wait before it has ELO > 3000 ? Maybe someone allows you to use a fast cluster.

You also don't know if you have chosen the right feature representation. Or not ?

I don't think you know what unsupervised learning means.

It's much more than "start the tuner and sit and wait". You go try that and see how well that works.

I had access to a 13000 cores cluster. I was only using about 100 CPUs. Faster cluster wouldn't help.

[Robert Pope]

This just in: http://www.technologyreview.com/view/54 ... al-master/

That looks to be a bit of a stretch to me. "at the level of top programs"? "at the level of IM"? First, those are contradictory terms. Second, I am not sure it is an IM yet, at least watching on ICC recently.

If you ever read a general news article about a technical issue that you know well, they almost never get it all right, unfortunately.

[bob]

This just in: http://www.technologyreview.com/view/54 ... al-master/

That looks to be a bit of a stretch to me. "at the level of top programs"? "at the level of IM"? First, those are contradictory terms. Second, I am not sure it is an IM yet, at least watching on ICC recently.

CCRL 40/40 after 189 games:
Giraffe 20150828 64-bit 2382 +42 −42 56.3% −42.6 31.2% 189

Wikipedia says:
International Master (IM)
"International Masters" redirects here. For the snooker tournament formerly known under this name, see British Open (snooker).

The title International Master is awarded to strong chess players. Instituted in 1950, it is a lifetime title, usually abbreviated as IM in chess literature.

Normally three norms in international tournaments involving other IMs and Grandmasters are required before FIDE will confer the title on a player. IMs usually have an Elo rating between 2400 and 2500. Sometimes, though, there may be an IM who has not yet become a Grandmaster but has a rating greater than 2500.

The IM title can also be awarded for a few specific performances. For example, under current rules, the runner up at the World Junior Championship will be awarded the IM title if he or she does not already have it. Current regulations may be found in the FIDE handbook.[6]

After becoming an IM, most professional players set their next goal as becoming a Grandmaster. It is also possible to become a Grandmaster without ever having been an International Master. Larry Christiansen of the United States (1977), Wang Hao of China, Anish Giri of The Netherlands, and former World Champion Vladimir Kramnik of Russia all became Grandmasters without ever having been an IM. Bobby Fischer of the United States attained both titles solely by virtue of qualifying for the 1958 Interzonal (IM title) and 1959 Candidates Tournament (GM title) entirely bypassing the usual process of achieving norms at each level only incidentally becoming IM before GM. However, the more usual path is first to become an IM, then move on to the GM level.

An International Master is usually in the top 0.25% of all tournament players at the time he or she receives the title.[7] The November 2010 FIDE rating list records 3036 players holding the IM title.[5]

No matter how you look at it, this is a very interesting project.
There are not many serious attempts to do chess evaluation based on learning and most attempts are not terribly successful.

Yes, I think the learning is a pretty interesting idea, since this might represent the first such working example in almost 50 years of computer chess experiments with learning.

But "IM" is a bit of a stretch so far, as is the "near the top programs". 2300 is not exactly near the 3400+ top programs... So Elo is a bit low for an IM (IMHO), and the "near the top" is something we all want to be, but aren't even close (Giraffe or Crafty or most anyone else except for a very rarified group).

[bob]

This just in: http://www.technologyreview.com/view/54 ... al-master/

That looks to be a bit of a stretch to me. "at the level of top programs"? "at the level of IM"? First, those are contradictory terms. Second, I am not sure it is an IM yet, at least watching on ICC recently.

In my paper I only claimed that the evaluation function is at the level of top programs, which I believe is substantiated, given that it scores about the same as Crafty on the STS, searching less than 1/10 as many nodes (and much higher with time odds so that it searches as many nodes).

It scores higher than Texel, at equal time (searching about 1/4 as many nodes).

I think they mis-interpreted it to cover the whole program.

The article was written without my input at all. I actually didn't find out about it until a few hours ago.

Crafty is not exactly "at the top level" either. Time odds doesn't mean much since chess is a timed game... Give me enough time and Crafty can beat StockFish.

[matthewlai]

This just in: http://www.technologyreview.com/view/54 ... al-master/

That looks to be a bit of a stretch to me. "at the level of top programs"? "at the level of IM"? First, those are contradictory terms. Second, I am not sure it is an IM yet, at least watching on ICC recently.

In my paper I only claimed that the evaluation function is at the level of top programs, which I believe is substantiated, given that it scores about the same as Crafty on the STS, searching less than 1/10 as many nodes (and much higher with time odds so that it searches as many nodes).

It scores higher than Texel, at equal time (searching about 1/4 as many nodes).

I think they mis-interpreted it to cover the whole program.

The article was written without my input at all. I actually didn't find out about it until a few hours ago.

Crafty is not exactly "at the top level" either. Time odds doesn't mean much since chess is a timed game... Give me enough time and Crafty can beat StockFish.

I agree, and that's made clear in the thesis as well, but not the article.

[MikeB]

This one only has some small changes. No 100+ Elo jump like last time.

https://bitbucket.org/waterreaction/gir ... 150908.zip

Changes:
1) Score reporting is now scaled down by 1/10. Hopefully that will play better with score-based adjudication systems. Score magnitudes are now more inline with what other engines report, though it's still probabilistic.
2) Time management has been improved (thanks to Graham Banks' testing that led me to notice this). About 33 Elo gained.
3) Gaviota tablebases now supported. See https://bitbucket.org/waterreaction/giraffe for details. I want to switch to a neural-network based endgame system later, but EGTB will do for now.

By the way, Giraffe is now open source, and the thesis is now available online. See thread on the programming forum if interested - http://talkchess.com/forum/viewtopic.ph ... 136#640136

This is probably going to be the last Giraffe release for a while. I have just submitted and defended my thesis. I think my brain needs a break from working on Giraffe full time for 3 months.

Will be travelling for a few weeks before getting back to Giraffe.

Thanks

Matthew,

Very interesting project and good luck with your thesis. I did compile a native Mac x86-64 bit version and played a short match against current developmental Crafty using a random , equal, alternating color book.

Code: Select all

ResultSet-EloRating>mm
00:00:00,00
ResultSet-EloRating>ratings
Rank Name             Elo    +    - games score oppo. draws 
   1 Crafty 25.0_TR   144  160  160    20   88%  -144    5% 
   2 Giraffe         -144  160  160    20   13%   144    5% 
ResultSet-EloRating>

Current Crafty is testing out at in the 3000-3050 range - so this is actually a very good result - as limited as it is with only 20 games. Giraffe saw 200 nps consistently on my machine, and occasionally spiked to around 400K/nps (x5690 i7 3.46 Ghz)

[kinderchocolate]

Congratulation for such an amazing accomplishment. I've read your thesis, and I think this is the best thesis/paper I've read for a while. Your project is skyrocketing, as I've been getting messages about your project from casual chess players.

I'll study your source code later. If possible, I want to try to train the model myself as I have the resources. Do you have documentation how I can train it?

[Vinvin]

I just see that : http://www.technologyreview.com/view/54 ... al-master/

[Henk]

With teaches itself I think about unsupervised learning. But it is not.

Neural networks are not much fun for you start the tuner and sit and wait.
Quite boring or not ? I don't like tuning at all.

But I would like to know the limits of this neural network approach. How long do you have to wait before it has ELO > 3000 ? Maybe someone allows you to use a fast cluster.

You also don't know if you have chosen the right feature representation. Or not ?

I don't think you know what unsupervised learning means.

It's much more than "start the tuner and sit and wait". You go try that and see how well that works.

I had access to a 13000 cores cluster. I was only using about 100 CPUs. Faster cluster wouldn't help.

Yes. I thought unsupervised learning is same as clustering.

If examples are not automatically generated NN learning is too much work if there are many parameters.

It is all about getting the right input and output. If you choose the wrong features NN will fail.

[MikeB]

Matthew,

Very interesting project and good luck with your thesis. I did compile a native Mac x86-64 bit version and played a short match against current developmental Crafty using a random , equal, alternating color book.

Code: Select all

ResultSet-EloRating>mm
00:00:00,00
ResultSet-EloRating>ratings
Rank Name             Elo    +    - games score oppo. draws 
   1 Crafty 25.0_TR   144  160  160    20   88%  -144    5% 
   2 Giraffe         -144  160  160    20   13%   144    5% 
ResultSet-EloRating>

Current Crafty is testing out at in the 3000-3050 range - so this is actually a very good result - as limited as it is with only 20 games. Giraffe saw 200 nps consistently on my machine, and occasionally spiked to around 400K/nps (x5690 i7 3.46 Ghz)

I did have a chance to watch Giraffe play a few of these games. My first thought - without hesitation , is that Giraffe appears to have very solid evaluation - I assume that's from being NN. It may be possible to incorporate that in traditional engines. because of it's solid evaluation and relative slow search, I suspected that it may test slightly better at faster time controls than slower time controls. The above 20 game set was ran at 1/1. Today I ran a 200 game set at 3/5 .

Code: Select all

ResultSet-EloRating>ratings
Rank Name             Elo    +    - games score oppo. draws 
   1 Crafty 25.0_TR   177   60   60   200   89%  -177    9% 
   2 Giraffe         -177   60   60   200   11%   177    9% 

A little bit worse, not really significant.

My favorite part about Giraffe is that plays it very human like moves. More so than any other engine I have seen. May not be always be the best move - but certainly it would be a good sparring partner for most of us and you can set it to play very fast.

This a game where both sides played without a book at 1/1.

[pgn][Event "Computer Chess Game"]
[Site "Michaels-Mac-Pro.local"]
[Date "2015.09.15"]
[Round "-"]
[White "Giraffe"]
[Black "Crafty 25.0_TR"]
[Result "0-1"]
[TimeControl "60+1"]
[Annotator "1. +0.12 1... +0.21"]

1. e4 {+0.12/15} e5 {+0.21/18 2.5} 2. c4 {+0.27/14 1.8} Bc5 {-0.21/17 2.5}
3. Nc3 {+0.25/15 3} Nf6 {-0.17/18 1.9} 4. Nf3 {+0.34/14 3} Ng4 {-0.15/20 5}
5. d4 {+0.03/15 1.9} exd4 {-0.15/18 1.8} 6. Nxd4 {+0.05/15 2.1} O-O
{-0.21/19 13} 7. h3 {+0.69/14 2.2} Nxf2 {-0.24/16 3} 8. Kxf2 {+1.49/16 2.0}
Qf6+ {-0.25/16 4} 9. Ke3 {+1.51/16 2.2} Nc6 {-0.25/15 1.4} 10. Ne2
{+1.61/15 1.8} Re8 {-0.54/14 1.4} 11. Rg1 {+1.30/13 2.1} Qe5 {-3.23/15 1.5}
12. Qd3 {-4.01/15 2.1} f5 {-3.22/15 1.3} 13. Rh1 {-4.66/14 1.7} fxe4
{-5.14/14 6} 14. Qd1 {-5.68/13 1.6} Nxd4 {-5.29/13 1.9} 15. Nxd4
{-6.43/14 2.0} Rf8 {-5.02/15 1.8} 16. Ke2 {-6.88/15 1.5} Bxd4
{-5.02/13 1.2} 17. Be3 {-6.89/15 2.5} c5 {-5.27/14 1.6} 18. Bxd4
{-7.25/14 2.1} cxd4 {-5.39/13 1.6} 19. Qb3 {-7.38/14 3} d3+ {-6.30/14 1.2}
20. Ke1 {-7.98/15 2.3} Qg3+ {-9.04/15 3} 21. Kd1 {-8.46/15 1.8} Qe3
{-8.52/18 1.5} 22. Bxd3 {-8.99/16 2.3} exd3 {-8.62/17 1.1} 23. Re1
{-9.22/15 1.6} Rf1 {-10.00/18 1.2} 24. Rxf1 {-9.30/18 2.1} Qe2+
{-10.00/16 1.1} 25. Kc1 {-9.31/18 2.2} d2+ {-10.00/15 1.1} 26. Kb1
{-9.30/17 2.8} Qxf1+ {-10.39/14 1.2} 27. Kc2 {-9.38/18 2.5} Qxa1
{-10.15/15 5} 28. c5+ {-9.40/17 2.7} Kh8 {-10.74/15 2.3} 29. Kxd2
{-9.42/16 1.4} b6 {-10.74/13 1.0} 30. Qd5 {-9.46/14 2.4} Qxb2+
{-13.52/12 1.0} 31. Kd1 {-9.56/15 2.2} Bb7 {-13.65/12 1.0} 32. Qxb7
{-9.73/14 2.6} Rf8 {-15.26/11 1.8} 33. Qa6 {-9.81/14 1.4} b5
{-327.52/14 2.8} 34. Qxb5 {-9.93/13 2.5} Qxb5 {-327.56/14 0.9} 35. Kd2
{-9.95/12 1.7} Re8 {-327.58/13 0.2} 36. g4 {-9.94/11 1.8} Qc4
{-327.62/9 0.1} 37. a4 {-29.99/11 2.5} Re2+ {-327.64/7 0.1} 38. Kd1
{-29.99/12 1.5} Qc2# {-327.66/5 0.1}
{Xboard adjudication: Checkmate} 0-1[/pgn]

Here's a game played at TC 3/5 with an opening position. As an FYI, Crafty won from the black side in about 50 less moves. May not be a truly equal position.

[pgn][Event "Computer Chess Game"]
[Site "Michaels-Mac-Pro.local"]
[Date "2015.09.15"]
[Round "14"]
[White "Crafty 25.0_TR"]
[Black "Giraffe"]
[Result "0-1"]
[TimeControl "180+5"]
[FEN "1r1q1rk1/2pbppbp/p1np1np1/1p4B1/2PPP3/1PN2P2/P2QN1PP/2R1KB1R w K - 0 1"]
[SetUp "1"]

{--------------
. r . q . r k .
. . p b p p b p
p . n p . n p .
. p . . . . B .
. . P P P . . .
. P N . . P . .
P . . Q N . P P
. . R . K B . R
white to play
--------------}
1. d5 {+0.24/20} b4 {+0.22/17 10} 2. dxc6 {+0.48/22 9} bxc3 {+0.06/18 13}
3. Nxc3 {+0.48/20 7} Bxc6 {+0.10/17 12} 4. Be2 {+0.41/20 47} Ba8
{-0.06/15 8} 5. c5 {+0.62/18 11} dxc5 {+0.48/16 11} 6. Na4 {+0.95/18 8} Nd7
{+0.52/17 9} 7. Be3 {+1.03/18 6} Qc8 {+0.33/16 13} 8. O-O {+1.14/20 6} Rd8
{-0.66/16 9} 9. Rfd1 {+1.10/20 6} Bb7 {-1.10/16 12} 10. Qa5 {+1.12/18 7}
Be5 {-1.20/16 7} 11. f4 {+1.42/20 6} Bg7 {-1.62/17 10} 12. e5 {+1.38/20 6}
e6 {-1.44/17 13} 13. Nxc5 {+1.34/20 8} Nxc5 {-1.46/18 9} 14. Qxc5
{+1.35/20 11} Rxd1+ {-1.25/18 13} 15. Bxd1 {+1.28/20 5} c6 {-1.08/17 11}
16. Kf2 {+1.33/20 12} Bf8 {-0.77/16 7} 17. Qa5 {+1.35/19 5} h5
{-0.94/16 10} 18. Be2 {+1.28/19 9} Kh7 {-0.97/15 12} 19. g3 {+1.43/19 9} c5
{+0.00/17 7} 20. Qb6 {+1.36/21 5} Bd5 {+0.00/18 11} 21. Qa7 {+1.36/19 5}
Rb7 {+0.13/17 12} 22. Qxa6 {+1.50/19 5} Qc7 {-0.11/17 6} 23. Qd3
{+1.53/19 9} Ra7 {-0.03/16 7} 24. a4 {+1.37/19 15} Rb7 {-0.22/17 10} 25.
Rc3 {+1.37/17 5} Rb4 {+0.18/16 11} 26. Qd1 {+1.23/19 6} Qb6 {-0.02/16 10}
27. Qa1 {+1.17/18 10} Bxb3 {+0.74/16 6} 28. a5 {+1.20/19 4} Qc6
{+1.06/17 8} 29. Qc1 {+1.32/20 7} c4 {+1.75/16 6} 30. Rxb3 {+0.76/25 11}
Rxb3 {+1.33/19 6} 31. Qxc4 {+0.71/24 4} Qxc4 {+1.04/20 9} 32. Bxc4
{+0.75/27 7} Rc3 {+1.36/20 7} 33. Bf1 {+0.57/29 21} Bc5 {+1.24/18 6} 34.
Bxc5 {+0.57/27 4} Rxc5 {+1.38/18 5} 35. a6 {+0.54/26 6} Rc3 {+1.69/18 3}
36. Ke2 {+0.38/23 11} Rc1 {+1.42/19 6} 37. Bg2 {+0.54/23 6} Rc2+
{+2.54/19 5} 38. Kf3 {+0.46/23 4} Kg7 {+2.63/20 5} 39. Bf1 {+0.29/23 5} Kf8
{+2.89/18 5} 40. Ke4 {+0.18/22 15} Ke8 {+2.50/17 3} 41. Bb5+ {+0.00/23 7}
Kd8 {+2.90/18 6} 42. h3 {+0.00/25 4} Rb2 {+3.16/18 2.8} 43. Bd3
{+0.00/25 4} Kc7 {+3.24/19 7} 44. Kd4 {+0.00/25 4} Kb6 {+3.24/19 8} 45. Ke3
{+0.00/28 4} Rg2 {+3.24/18 4} 46. Kf3 {+0.00/28 5} Rh2 {+3.24/17 2.8} 47.
h4 {+0.00/29 4} Rd2 {+3.24/18 6} 48. Be2 {+0.00/29 4} Ra2 {+3.26/20 3} 49.
Ke3 {+0.00/29 5} Ra3+ {+3.26/22 6} 50. Bd3 {+0.00/33 4} Kc5 {+3.26/22 4}
51. Ke4 {+0.00/25 4} Ra4+ {+3.26/21 10} 52. Ke3 {+0.00/36 4} Ra5
{+3.25/20 4} 53. Be2 {+0.00/34 4} Ra3+ {+4.19/19 3} 54. Bd3 {+0.00/39 4}
Kd5 {+5.98/19 2.7} 55. Ke2 {-1.93/26 6} Kd4 {+6.10/20 3} 56. Bb5
{-1.91/26 8} Rxg3 {+6.08/21 10} 57. Be8 {-1.93/25 6} Re3+ {+6.04/20 5} 58.
Kf2 {-1.98/25 6} Ra3 {+6.26/19 2.8} 59. Bxf7 {-2.32/24 6} Ke4 {+6.30/20 3}
60. Bxe6 {-2.28/24 8} Kxf4 {+6.64/21 8} 61. Bd7 {-2.31/24 11} Ra2+
{+6.95/17 2.8} 62. Kf1 {-2.32/25 10} Kxe5 {+7.79/20 8} 63. Bb5 {-2.40/25 3}
Kf4 {+7.96/20 4} 64. Bc4 {-4.41/23 8} Ra1+ {+8.10/19 3} 65. Kf2
{-5.62/22 8} Kg4 {+8.12/18 3} 66. Kg2 {-5.91/23 7} Kxh4 {+8.62/19 6} 67.
Bb5 {-6.83/22 6} Ra2+ {+9.01/20 7} 68. Kf1 {-7.17/22 5} Kg3 {+9.09/18 3}
69. Bc4 {-8.97/23 5} Ra1+ {+9.52/19 6} 70. Ke2 {-8.97/10} h4 {+9.53/19 4}
71. Kd3 {-8.97/20 8} h3 {+9.60/19 9} 72. Kc2 {-9.69/18 6} h2 {+9.78/19 4}
73. Bd5 {-9.20/15 6} Rxa6 {+9.80/18 4} 74. Kd3 {-327.25/16 5} Rd6
{+9.84/16 3} 75. Kc4 {-327.47/31 3} Rxd5 {+9.88/16 9} 76. Kxd5
{-327.49/23 0.3} h1=Q+ {+9.90/15 4} 77. Kd6 {-327.51/23 0.6} g5
{+9.93/14 5} 78. Kc5 {-327.51/23 0.3} Kf4 {+9.96/15 4} 79. Kd6
{-327.53/23 0.3} Qb7 {+29.98/15 4} 80. Kc5 {-327.55/23 0.2} g4
{+29.98/15 5} 81. Kd6 {-327.57/23 0.1} g3 {+29.99/14 5} 82. Kc5
{-327.59/23} g2 {+29.99/14 5} 83. Kc4 {-327.61/23} g1=Q {+29.99/14 5}
{White resigns} 0-1[/pgn]

[d]5b2/1rq2p1k/4p1p1/2pbP2p/5P2/1P1QB1P1/P3BK1P/2R5 b - - 2 23

On the 23rd move - White is looking pretty good, a pawn up +1.5 - but it was all downhill from here. But Giraffe appears to know that things are not as bad as they look and its score reflects a more balanced game.

[d] 5b2/5p1k/2q1p1p1/P3P2p/1rp2P2/1bR1B1P1/4BK1P/2Q5 w - - 0 30

Developmental crafty gives up the exchange to grab that pesky pawn - but it's also a loss position and Giraffe plays like a chess master technician from this point forward to win the game.