TalkChess.com

Does anybody know of a database program that understands Shredder FENs? I am trying to extract some information from the CCRL FRC database, but I am having little luck. Unfortunately, Scid and its offshoots choke on the Shredder FEN strings found in the PGNs.

Thanks,
Adam

It depends on what information you want. If it is very basic (like how many times a certain opening position occured, and how often it was won for white) you could us WinBoard. I suppose the FRC database is not that huge.

hgm wrote:It depends on what information you want. If it is very basic (like how many times a certain opening position occured, and how often it was won for white) you could us WinBoard. I suppose the FRC database is not that huge.

Unfortunately, the information required is about castling.

Scid and PGN Extract do not understand the rook location token(?) (HAha, HBhb, ... GAga, ..., etcetera). When I modify the FEN string so that HAha et al is replaced with KQkq, Scid drops its objections to the FEN tag, but it still does not understand FRC castling. I also tried CQL (Chess Query Language), but it does not seem to recognize FEN tags (the PGN it produces from a query lacks a FEN tag and it does not recognize when pieces are in non-standard positions).

Well, so what do you want to know? WinBoard is open source, so there is no limit to what it could be made to do. There already is code in there to parse all games (doesn't matter whether is is FRC, Xiangi or Gothic Chess), and store the moves in a compressed form in memory. I am sure it would be trivial to take some stats in the process, and let it pop up a window too present it when it completes a search command.

hgm wrote:Well, so what do you want to know? WinBoard is open source, so there is no limit to what it could be made to do. There already is code in there to parse all games (doesn't matter whether is is FRC, Xiangi or Gothic Chess), and store the moves in a compressed form in memory. I am sure it would be trivial to take some stats in the process, and let it pop up a window too present it when it completes a search command.

I have yet to find a limitation that did not ultimately lead back to me, rather than Winboard

There has been some interest as to how often same-side and opposite-side castling occurs in the CCRL FRC database. As well as examining white score for each starting position, where the opponents are above a specified Elo and are within a specified Elo difference. I have found a hack that will allow me to examine the second area of interest, but not the first.

Are you entertaining the idea of expanding the database functionality (I hope that phrasing conveys my thought) of Winboard. If so, that would be great, but unexpected.

Hmm, actually this discussion turned up a bug: by inspecting the code in order to remind me how castlings are encoded, I noticed that they are performed in a way that could be incorrect for some FRC positions: first it clears the king square and puts the king on its destination, then it clears the rook square, and puts the rook on its destination. But the king could have ended up on the square where the rook started, which would give a nice disappearance act.

About the database functionality: within the current framework it is quite easy to write code that would answer any specific question (such as the one about castling you mentioned; if it were not for the bug I stumbled on, I could already have done it). But there are a lot of possible questions, and it is not really clear to me how one would make a method of specifying them general enough to accommodate them all. (But of course I have never looked at a real database program.) For instance, it never occurred to me that one would want to search for moves, rather than positions.

I could add a new search mode, 'find move', that would only select games which contain the move played to the currently displayed position (indicated by the highlights). But you want to select on two moves, one for each side. Of course this could be done by multi-step selection, allowing the user to limit the search to what is already selected, rather than searching all games on every query. (E.g. add a checkbox 'search selection')

It could also be useful to allow the user to 'export' the selection. Currently WinBoard can only save single games.

Hi Adam,

Forgive me if I don't quite understand your problem.

You have a PGN file, a database of many PGN Fischer Randon games, and you are interested in knowing how often each side castles on the same side, on opposite sides, etc. Yes?

A year or two ago I wrote a little utility for myself called "PGN Cleanup" which, among other things, gives a bunch of statistics on the file it processes. And yes, one of those statistics are a few castling statistics.

For example, below is the partial output from a PGN file that I downloaded and then processed:


Win / Loss Statistics:
----------------------
Number of games White won: 20,306 (38.4%)
Number of games Black won: 15,628 (29.6%)
Number of draws: 16,924 (32.0%)

Castling Statistics:
--------------------
White castled Kingside and Black castled Kingside: 35,932 (68.0%)
White castled Kingside and Black castled Queenside: 1,878 (3.6%)
White castled Kingside and Black did not castle: 3,920 (7.4%)

White castled Queenside and Black castled Kingside: 3,566 (6.7%)
White castled Queenside and Black castled Queenside: 1,020 (1.9%)
White castled Queenside and Black did not castle: 1,504 (2.8%)

White did not castle and Black castled Kingside: 2,694 (5.1%)
White did not castle and Black castled Queenside: 568 (1.1%)
Neither player castled: 1,774 (3.4%)


Even though your file is a file of FR games, with a Shredder FEN tag to distinguish the opening position, the castling notation within the PGN remains the same, does it not? O-O for Kingside Castling and O-O-O for Queenside castling? If so, my program wouldn't care if the games were FR games or normal games... it would generate a similar set of statistics for your file.

hgm wrote:Hmm, actually this discussion turned up a bug: by inspecting the code in order to remind me how castlings are encoded, I noticed that they are performed in a way that could be incorrect for some FRC positions: first it clears the king square and puts the king on its destination, then it clears the rook square, and puts the rook on its destination. But the king could have ended up on the square where the rook started, which would give a nice disappearance act.

Well, at least it was just the rook that disappeared. I know from experience that a program can function reasonably well if a piece occasionally vanishes. But, let the king disappear and you have total havoc.

Best
Dan H.

Adam Hair wrote:Unfortunately, the information required is about castling.

Scid and PGN Extract do not understand the rook location token(?) (HAha, HBhb, ... GAga, ..., etcetera).

There is a scid960 project. Unfortunately the last i heard from the person was that he had found bugs, and was going to do some more work on it, but have had no follow up.

I have a working patch for Scid vs PC, if anyone's interesting in testing it and telling me what works and what doesn't. Perhaps we can make it work properly ? Shredder FEN seem to be working ok.

hgm wrote:Hmm, actually this discussion turned up a bug: by inspecting the code in order to remind me how castlings are encoded, I noticed that they are performed in a way that could be incorrect for some FRC positions: first it clears the king square and puts the king on its destination, then it clears the rook square, and puts the rook on its destination. But the king could have ended up on the square where the rook started, which would give a nice disappearance act.

At least something good happened due to my question

hgm wrote: About the database functionality: within the current framework it is quite easy to write code that would answer any specific question (such as the one about castling you mentioned; if it were not for the bug I stumbled on, I could already have done it). But there are a lot of possible questions, and it is not really clear to me how one would make a method of specifying them general enough to accommodate them all. (But of course I have never looked at a real database program.) For instance, it never occurred to me that one would want to search for moves, rather than positions.

I could add a new search mode, 'find move', that would only select games which contain the move played to the currently displayed position (indicated by the highlights). But you want to select on two moves, one for each side. Of course this could be done by multi-step selection, allowing the user to limit the search to what is already selected, rather than searching all games on every query. (E.g. add a checkbox 'search selection')

It could also be useful to allow the user to 'export' the selection. Currently WinBoard can only save single games.

I would think that most often the focus would be on a position, rather than a move.

I have to agree that it is very tough trying to anticipate all (or most) possible questions. I guess this is a matter of determining what things would be useful to have in Winboard. If we were only talking about orthodox chess, then I would think the existing functionality is or is close to being sufficient. After all, there are several programs dedicated to working with chess databases. However, if we take into account variants of chess (and other games), then there is nothing I know of that can do the job. Whether or not Winboard should fill that gap seems to depend on you at the moment.