I think the entire idea is to lure me into one of your fights, won't happen, but keep on trying
Syzygy bases from memory
Moderators: hgm, Rebel, chrisw
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: Syzygy bases from memory
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 27790
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Syzygy bases from memory
OK, so your idea is that this is a fight. That explains a lot.
I, however, are interested in chessprogramming facts. And it should be clear to all readers now that for perfect evaluation of draws in end-games that have on the order of 10% (or 90%), by far the best method is just store the draw/non-draw bitbase in memory, and probe the relevant bit by a simple memory access. Other methods are of course possible, but they require many times the amount of memory, and are easily an order of magnitude slower.
Especially just storing a table of only the draws (or wins) is not competative, because 10% is just not few enough to earn back the position encoding this requires.
Don't be fooled by persons who claim the contrary and are evasive when asked for details or throw smoke. They are wrong, and they know it.
I, however, are interested in chessprogramming facts. And it should be clear to all readers now that for perfect evaluation of draws in end-games that have on the order of 10% (or 90%), by far the best method is just store the draw/non-draw bitbase in memory, and probe the relevant bit by a simple memory access. Other methods are of course possible, but they require many times the amount of memory, and are easily an order of magnitude slower.
Especially just storing a table of only the draws (or wins) is not competative, because 10% is just not few enough to earn back the position encoding this requires.
Don't be fooled by persons who claim the contrary and are evasive when asked for details or throw smoke. They are wrong, and they know it.
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: Syzygy bases from memory
I looked into your code, saw it came under the UBC license (Use But Credit) and wondered if I could use it, did some test positions and it did very well, is it 100% accurate? But my compiler could not compile it and I gave up. Regarding KPKQ, I prune too, white can only draw if the pawn is at least at the 6th row and the white king is close to the pawn, and if the black queen is not on the promotion square.jonkr wrote: ↑Sat Jun 26, 2021 6:00 am The endgame neural nets in SlowChess are primarily for board positions larger than 5-man usually multiple pieces and any number of pawns. They are trained on W/L/D result of the sample games so that is similar (though eventually started blending in some search scoring after initial training on some of them.) I do think training a neural net to replace a tablebase could do pretty well for a much smaller size, probably well enough that the difference would be hard to measure. But can't beat perfect, I'm sure I'd end up seeing some loss like maybe in TCEC if I dropped tablebases then want to re-add support.
I plan on trying to use distance-to-mate in the target value to train a mating net (for no-pawns positions) to replace my hard-coded MatingEval function, to see if it can find mates quicker, just as an experiment. But probably won't try specific endgames with only a few pieces.
The SlowChess bitbase for KP-KR is 428 kb, it only stores a win/draw bit. I would have to look to see how else it's stored, my bitbases are from 2005 I think haha. I think the syzygy for that endgame are actually smaller though? I was surprised at how small the syzygy were when I first looked at them.
I remember for some like KPKQ I had some piece position checks and only stored possibly drawn positions so that one is only 40kb.
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: Syzygy bases from memory
I am not in the mood to rehash my goals, Chris kindly remembered you, you choose to be deaf. The choice for a TT is because a) I don't need bit-based for my goal and b) the TT approach outperforms bit-bases because I can use it in search as well.hgm wrote: ↑Sat Jun 26, 2021 11:31 am OK, so your idea is that this is a fight. That explains a lot.
I, however, are interested in chessprogramming facts. And it should be clear to all readers now that for perfect evaluation of draws in end-games that have on the order of 10% (or 90%), by far the best method is just store the draw/non-draw bitbase in memory, and probe the relevant bit by a simple memory access. Other methods are of course possible, but they require many times the amount of memory, and are easily an order of magnitude slower.
Especially just storing a table of only the draws (or wins) is not competative, because 10% is just not few enough to earn back the position encoding this requires.
Don't be fooled by persons who claim the contrary and are evasive when asked for details or throw smoke. They are wrong, and they know it.
Now, you may have the last word, be kind for a change.
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 27790
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Syzygy bases from memory
I would like to be 'kind', but if you write nonsense, it is a tad hard to remain truthful and call it anything but nonsense... This is a chessprogramming forum with many readers, so it is important that info exhibited here is factually correct and not misleading. And if you consider it unkind to be caught at an error, then so be it.
For one, what 'TT approach'? In usual chessprogramming jargon TT means Transposition Table, and it is the first time you mention it in this respect. (And not clear how it would be useful in this context either.) In a previous posting you said that you 'hashed' the drawn positions (presumably meaning converting the position to a unique number, not necessarily storing it in a hash table), and using a binary search to find back the key (similar to how Polyglot implements its opening book).
As I and several others pointed out: this is bulky (because the keys are many bits, not 1) and slow (because binary search needs log(N) ~ 20 probes into the bulky set that doesn't fit in cache, rather than a single probe into a data set so small it might even fit in L3). Yet you keep insisting it is faster...
It is a complete mystery why you now suddenly claim bitbases cannot be used in search. Especially since this is what everyone who uses bitbases does. And it makes no sense: a bitbase probe gives you exactly the same information as the binary search, namely whether a given position is a draw or not. So of course you can use it for the same purpose, whatever that purpose may be. It is just nonsensical to claim otherwise.
For one, what 'TT approach'? In usual chessprogramming jargon TT means Transposition Table, and it is the first time you mention it in this respect. (And not clear how it would be useful in this context either.) In a previous posting you said that you 'hashed' the drawn positions (presumably meaning converting the position to a unique number, not necessarily storing it in a hash table), and using a binary search to find back the key (similar to how Polyglot implements its opening book).
As I and several others pointed out: this is bulky (because the keys are many bits, not 1) and slow (because binary search needs log(N) ~ 20 probes into the bulky set that doesn't fit in cache, rather than a single probe into a data set so small it might even fit in L3). Yet you keep insisting it is faster...
It is a complete mystery why you now suddenly claim bitbases cannot be used in search. Especially since this is what everyone who uses bitbases does. And it makes no sense: a bitbase probe gives you exactly the same information as the binary search, namely whether a given position is a draw or not. So of course you can use it for the same purpose, whatever that purpose may be. It is just nonsensical to claim otherwise.
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: Syzygy bases from memory
Ah, you could not, not a real surprise.
Once again and last -
I DON'T NEED BIT-BASES
Now shut up and leave me alone.
Once again and last -
I DON'T NEED BIT-BASES
Now shut up and leave me alone.
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 27790
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Syzygy bases from memory
It is not relevant what you need or don't need. The world doesn't revolve around you. What is important is that the readers can learn the best way to solve the indicated problem here. If you want to show them how better not to do that, that's fine too.
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: Syzygy bases from memory
It's not a matter what is best, you judge and you have no right. I am perfectly fine with the chosen format as it contains the best move for free to guide the search something bit-bases don't have. No loss in NPS, a binary search in memory is peanuts. Within a couple of days I will release the positions of KPK, KRKP, KQKP, KRKN and KRKB with Syzygy score and best move in EPD format. Anyone interested can create their own chosen format, whatever that suits them best. There is no absolute truth in this matter, you claim it, utter nonsense.
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 5557
- Joined: Tue Feb 28, 2012 11:56 pm
Re: Syzygy bases from memory
It should all be in the system cache in RAM after the first probe and stay there. It's not slow (especially if you are just probing through it once to find the holes).Rebel wrote: ↑Wed Jun 16, 2021 11:22 am I want to fix some holes in my static KPvK evaluation from the 80's. It catches most (80-90%) of the winning positions but not all. Installing the Syzygy bases is of course the most simple solution but it is slow because it runs from HD. The question is if it can run from memory and if loading "KPvK.rtbw" and "KPvK.rtbz" at program start is sufficient.
-
- Posts: 5557
- Joined: Tue Feb 28, 2012 11:56 pm
Re: Syzygy bases from memory
If you want the TBs to be in RAM during probing, forget about RAM drives. They just take memory away from the RAM available for the system cache. (Unless you have a smart RAM drive implementation, but I doubt those exist on Windows.)