Crafty 25.0 Release

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
MikeB
Posts: 3467
Joined: Thu Mar 09, 2006 5:34 am
Location: Pen Argyl, Pennsylvania

Crafty 25.0 Release

Post by MikeB » Sat Dec 26, 2015 2:24 am

Image
We are please to announce the release of Crafty 25.0 (source) for your enjoyment. We will soon have links posted with the appropriate exe's.

A merry Christmas and joyous holiday season to all!

Best wishes,
Robert Hyatt, University of Alabama at Birmingham
Michael Byrne, Pen Argyl, PA
Tracy Riegle, Hershey, PA
Peter Skinner, Edmonton, AB Canada

Code: Select all

Crafty 25.0 

*  This version contains a major rewrite of the parallel search code, now referred to as Generation II. It has a more lightweight split algorithm, that costs the parent MUCH less effort to split the search. The key is that now the individual "helper" threads do all the work, allocating a split block, copying the data from the parent, etc., rather than the parent doing it all. Gen II based on the DTS "late-join" idea so that a thread will try to join existing split points before it goes to the idle wait loop waiting for some other thread to split with it. In fact, this is now the basis for the new split method where the parent simply creates a split point by allocating a split block for itself, and then continuing the search, after the parent split block is marked as "joinable". Now any idle threads just "jump in" without the parent doing anything else, which means that currently idle threads will "join" this split block if they are in the "wait-for-work" spin loop, and threads that become idle also join in exactly the same way. This is MUCH more efficient, and also significantly reduces the number of split blocks needed during the search. We also now pre-split the search when we are reasonably close to the root of the tree, which is called a "gratuitous split. This leaves joinable split points laying around so that whenever a thread becomes idle, it can join in at these pre-existing split points immediately. We now use a much more conservative approach when dealing with fail highs at the root. Since LMR and such have introduced greater levels of search instability, we no longer trust a fail-high at the root if it fails low on the research. We maintain the last score returned for every root move, along with the PV. Either an exact score or the bound score that was returned. At the end of the iteration, we sort the root move list using the backed-up score as the sort key, and we play the move with the best score. This solves a particularly ugly issue where we get a score for the first move, then another move fails high, but then fails low and the re-search produces a score that is actually WORSE than the original best move. We still see that, but we always play the best move now. One other efficiency trick is that when the above happens, the search would tend to be less efficient since the best score for that fail-high/fail-low move is actually worse than the best move/score found so far. If this happens, the score is restored to the original best move score (in Search()) so that we continue searching with a good lower bound, not having to deal with moves that would fail high with this worse value, but not with the original best move's value. 

*  We also added a new method to automatically tune the new SMP parameters. The command is autotune and "help autotune" will explain how to run it. 

*  In addition , we did a complete re-factor of pawn evaluation code. There were too many overlapping terms that made tuning difficult. Now a pawn is classified as one specific class, there is no overlap between classes, which simplifies the code significantly. The code is now easier to understand and modify. In addition, the passed pawn evaluation was rewritten and consolidates all the passed pawn evaluation in one place. The evaluation used to add a bonus for rooks behind passed pawns in rook scoring, blockading somewhere else, etc. All of this was moved to the passed pawn code to make it easier to understand and modify. 

*  Added a limited version of late move pruning (LMP) for the last two plies. Once a set number of moves have been searched with no fail high, non-interesting moves are simply skipped in a way similar to futility pruning. 

*  We had a minor change to history counters that now rely on a "saturating counter" idea. I wanted to avoid the aging idea, and it seemed to not be so clear that preferring history moves by the depth at which they were good was the way to go. I returned to a history counter idea I tested around 2005 but discarded, namely using a saturating counter. The idea is that a center value (at present 1024) represents a zero score. Decrementing it makes it worse, incrementing it makes it better. But to make it saturate at either end, I only reduce the counter by a fraction of its distance from the saturation point so that once it gets to either extreme value, it will not be modified further avoiding wrap-around. This basic idea was originally reported by Mark Winands in 2005. It seems to provide better results (slightly) on very deep searches. One impetus for this was an intent to fold this into a move so that I could sort the moves rather than doing the selection approach I currently use. However, this had a bad effect on testing, since history information is dynamic and is constantly changing, between two moves at the same ply in fact. The sort fixed the history counters to the value at the start of that ply. This was discarded after testing, but the history counter based on the saturating counter idea seemed to be OK and was left in even though it produced minimal Elo gain during testing. 

*  We change to the way moves are counted, to add a little more consistency to LMR. Now Next*() returns an order number that starts with 1 and monotonically increases, this order number is used for LMR and such decisions that vary things based on how far down the move list something occurs. Root move counting was particularly problematic with parallel searching, now things are at least "more consistent". The only negative impact is that now the move counter gets incremented even for illegal moves, but testing showed this was a no-change change with one thread, and the consistency with multiple threads made it useful. 

*  Added the "counter-move" heuristic for move ordering (Jos Uiterwijk, JICCA) which simply remembers a fail high move and the move at the previous ply. If the hash, captures or killer moves don't result in a fail high, this move is tried next. No significant cost, seems to reduce tree size noticeably. Added a follow-up idea based on the same idea, except we pair a move that fails high with the move two plies back, introducing a sort of "connection" between them. This is a sort of "plan" idea where the first move of the pair enables the second move to fail high. The benefit is that this introduces yet another pair of good moves that get ordered before history moves, and is therefore not subject to reduction. I have been unable to come up with a reference for this idea, but I believe I first saw it somewhere around the time Fruit showed up, I am thinking perhaps in the JICCA/JICGA. Any reference would be appreciated. 

*  A minor change to the way the PV and fail-hi/fail-low moves are displayed when pondering. 

*  Crafty now adds the ponder move to the front of the PV enclosed in parentheses so that it is always visible in console mode. The depths are reaching such extreme numbers the ponder move scrolls off the top of the screen when running in console mode or when "tail -f" is used to watch the log file while a game is in progress. This is a bit trickier than you might think since Crafty displays the game move numbers in the PV. 

*  The penalty for pawns on same color as bishop now only applies when there is one bishop.

schack
Posts: 136
Joined: Thu May 27, 2010 1:32 am
Contact:

Re: Crafty 25.0 Release

Post by schack » Sat Dec 26, 2015 3:18 am

I know that Bob will never go the route of UCI, but I know Chessbase used to make native CB versions of Crafty available. Is there any way that might begin again?

Thanks much for all the work, all of you!

MikeB
Posts: 3467
Joined: Thu Mar 09, 2006 5:34 am
Location: Pen Argyl, Pennsylvania

Re: Crafty 25.0 Release

Post by MikeB » Sat Dec 26, 2015 4:17 am

For the Mac - xBoard engine - 4 flavors in a 1MB download:

Latest Macs
Later Macs
Most Macs
No EGTB

Crafty 25.0 Mac Exe's

You may have to change permissions:

'chmod 755 <filename>' for the GUI to recognize the app.

bob
Posts: 20558
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Crafty 25.0 Release

Post by bob » Sat Dec 26, 2015 4:50 am

schack wrote:I know that Bob will never go the route of UCI, but I know Chessbase used to make native CB versions of Crafty available. Is there any way that might begin again?

Thanks much for all the work, all of you!
I don't think they have the interest any longer...

For UCI, all I can suggest is wb2uci. But the protocol is flawed enough I'm simply not interested.

Frank Quisinsky
Posts: 4852
Joined: Wed Nov 18, 2009 6:16 pm
Location: Trier, Germany
Contact:

Re: Crafty 25.0 Release

Post by Frank Quisinsky » Sat Dec 26, 2015 9:58 am

Hi Bob,

I am in wait position with Crafty since I am reading that you are working hard for that release.

Together with BobCat 7.1 I will test Crafty 25 for the "Fight for League 2". I am thinking Crafty will make a big jump in my Rating List. Will be start December 29th, 2015 after the still running Dirty test-run ... again together with BobCat on 7 cores. Will be run around 10 days and I have a nice holiday. Like Bobcat and Crafty ...

Have thanks for your work again and good luck, nice Christmas for you too. I will have it with your work.

Best
Frank

And to the question UCI or WB?!
The first Crafty 24er versions have "lost on time" problems with WB2uci Adapter under Shredder Classic GUI. But you solved the problem with the latest 24er releases. Crafty runs perfect for me with Winboard protocol. All is fine here ... I think Crafty should hold the Winboard flag.
I like computer chess!

dchoman
Posts: 171
Joined: Wed Dec 28, 2011 7:44 pm
Location: United States

Re: Crafty 25.0 Release

Post by dchoman » Sat Dec 26, 2015 2:57 pm

MikeB wrote:

Code: Select all

*  Added the "counter-move" heuristic for move ordering &#40;Jos Uiterwijk, JICCA&#41; which simply remembers a fail high move and the move at the previous ply. If the hash, captures or killer moves don't result in a fail high, this move is tried next. No significant cost, seems to reduce tree size noticeably. Added a follow-up idea based on the same idea, except we pair a move that fails high with the move two plies back, introducing a sort of "connection" between them. This is a sort of "plan" idea where the first move of the pair enables the second move to fail high. The benefit is that this introduces yet another pair of good moves that get ordered before history moves, and is therefore not subject to reduction. I have been unable to come up with a reference for this idea, but I believe I first saw it somewhere around the time Fruit showed up, I am thinking perhaps in the JICCA/JICGA. Any reference would be appreciated. 
EXchess has used both approaches for a few years now. I wrote a thread asking about the second idea a few years ago, and it turned out others have tried it with different degrees of success. Here's the thread...

http://www.talkchess.com/forum/viewtopi ... highlight=

- Dan

bob
Posts: 20558
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Crafty 25.0 Release

Post by bob » Sat Dec 26, 2015 4:02 pm

dchoman wrote:
MikeB wrote:

Code: Select all

*  Added the "counter-move" heuristic for move ordering &#40;Jos Uiterwijk, JICCA&#41; which simply remembers a fail high move and the move at the previous ply. If the hash, captures or killer moves don't result in a fail high, this move is tried next. No significant cost, seems to reduce tree size noticeably. Added a follow-up idea based on the same idea, except we pair a move that fails high with the move two plies back, introducing a sort of "connection" between them. This is a sort of "plan" idea where the first move of the pair enables the second move to fail high. The benefit is that this introduces yet another pair of good moves that get ordered before history moves, and is therefore not subject to reduction. I have been unable to come up with a reference for this idea, but I believe I first saw it somewhere around the time Fruit showed up, I am thinking perhaps in the JICCA/JICGA. Any reference would be appreciated. 
EXchess has used both approaches for a few years now. I wrote a thread asking about the second idea a few years ago, and it turned out others have tried it with different degrees of success. Here's the thread...

http://www.talkchess.com/forum/viewtopi ... highlight=

- Dan
That might have been where I saw it, I simply didn't remember anything except the idea. I am using it for a slightly twisted idea. Not because it is a good ordering idea (it seems to add nothing to Crafty in that regard) but because it gives me another move to try before I get to the last stage of move generation where LMR, LMP, etc apply. Until I get to the "history" or "remaining" phases of the move selection process, no reductions and such are done. This was only worth a couple of Elo, but any positive change is good since my measurements are done via real games so that computational cost gets factored in...

Dirt
Posts: 2851
Joined: Wed Mar 08, 2006 9:01 pm
Location: Irvine, CA, USA

Re: Crafty 25.0 Release

Post by Dirt » Sat Dec 26, 2015 5:07 pm

Is there syzygy support? I don't see any, but I don't have that much faith in my powers of observation.
Deasil is the right way to go.

User avatar
Graham Banks
Posts: 33150
Joined: Sun Feb 26, 2006 9:52 am
Location: Auckland, NZ

Re: Crafty 25.0 Release

Post by Graham Banks » Sat Dec 26, 2015 6:02 pm

MikeB wrote:......We will soon have links posted with the appropriate exe's. ...
Excellent news. :)
My email addresses:
gbanksnz at gmail.com
gbanksnz at yahoo.co.nz

MikeB
Posts: 3467
Joined: Thu Mar 09, 2006 5:34 am
Location: Pen Argyl, Pennsylvania

Re: Crafty 25.0 Release

Post by MikeB » Sat Dec 26, 2015 6:38 pm

Dirt wrote:Is there syzygy support? I don't see any, but I don't have that much faith in my powers of observation.
Nalimov EGTB support.

Post Reply