PGN standard, its improvement and standardization

[sovaz1997]

Due to the fact that we have a huge number of different chess shells, I believe that it is necessary to create a new format for PGN files, which will standardize computer position evals, its analysis and additional data. With a single format, it will be much more convenient to use one PGN file in various chess shells.

For example we have TCEC comments format:

Code: Select all

{d=16, sd=51, mt=161000, tl=7049000, s=219832, n=35024512, pv=Nb3 Be7 h4 Rb8 h5 b5 Bd3 f6 exf6 Nxf6 Nd4 Na5 Qe2 Bb4 Kb1 Rb7 Nb3 Nc4 Bd4 Bd6 h6 g6 g3 Rff7 Bxc4, tb=0, h=86.3, ph=0.0, wv=1.08, R50=49, Rd=-11, Rr=-11, mb=+0+0+0+0+0,}

And tags:

Code: Select all

[Event "TCEC Season 16 - Superfinal"]
[Site "https://tcec-chess.com"]
[Date "2019.10.02"]
[Round "39.1"]
[White "AllieStein v0.5-dev_7b41f8c-n11"]
[Black "Stockfish 19092522"]
[Result "1-0"]
[BlackElo "3892"]
[ECO "C11"]
[GameDuration "03:50:43"]
[GameEndTime "2019-10-02T22:50:54.482 W. Europe Standard Time"]
[GameStartTime "2019-10-02T19:00:10.807 W. Europe Standard Time"]
[Opening "French"]
[PlyCount "120"]
[Termination "adjudication"]
[TerminationDetails "TCEC win rule"]
[TimeControl "7200+10"]
[Variation "Steinitz, Boleslavsky variation"]
[WhiteElo "3823"]

This is not standard. Different shells even in tags use different types of time recording (7200 + 10, 7200s + 10s).

Octagon GUI:

Code: Select all

[OS "Win10 x64"]
[CPU "i9-7960X CPU 32 Threads"]
[GPU "2 x GeForce RTX 2070"]
[Memory "64Gb"]
[TournamentFormat "Gauntlet"]
[WhiteTimeControl "7200s+10s"]
[BlackTimeControl "7200s+10s"]
[TimeOdds "1"]
[OptionsWhite ""]
[OptionsBlack ""]
[OpeningOptions ""]
[Tablebases ""]
[WinRule "1 moves, evals >=10.0 for 5 moves"]
[DrawRule "35 moves, evals <= 0.08 for 5 moves"]
[PlyCount "232"]
[Time "04:40:14433"]
[Termination "normal"]
[End Reason "White wins by adjudication rule"]

I suggest discussing on this topic: which tags really need to be standardized, which ones do not. In any case, this will be an extension of the PGN format, and not its replacement. I plan to write a standard, develop translators from different formats in PGN2 and write a test GUI that will show files of a new type.

[BeyondCritics]

Principally i endorse your brave efforsts, but it won't work. The most you can hope for, is dying a peacuful dead, without anybody noticing.
Why? Because there are a lots of commercial and private vendors who do rely on the pgn standard and if you are succesful in changing that standard -- even in a fully "backward compatible" manner -- you will cost them money. You will cost them money and that makes them very nervous.
If you want to have at least a slim chance to change something to the better, you should address that issue upfront. This is only my opinion. Good luck anyway.

[hgm]

Make sure to include the following tags:

* Variant
possible values e.g. suicide, atomic, capablanca, omega, xiangi, shogi, ...

* VariantMen
value is a comma-separated list of <piece id> : <Betza move descriptor> pairs for all unorthodox pieces participating in the game

[Vinvin]

Some tags from Lichess :

[Event "Rated Crazyhouse game"]
[Site "https://lichess.org/0E9HoiTK"]
[Date "2019.09.14"]
[Round "-"]
[White "chickencrossroad"]
[Black "gsvc"]
[Result "1-0"]
[UTCDate "2019.09.14"]
[UTCTime "00:34:12"]
[WhiteElo "2635"]
[BlackElo "2638"]
[WhiteRatingDiff "+7"]
[BlackRatingDiff "-8"]
[WhiteTitle "FM"]
[BlackTitle "IM"]
[Variant "Crazyhouse"]
[TimeControl "60+0"]
[ECO "?"]
[Opening "?"]
[Termination "Normal"]
[Annotator "lichess.org"]

1. e4 Nf6 2. Nc3 Nc6 3. Nf3 d5 4. exd5 Nxd5 5. Bc4 Nxc3 6. Bxf7+ Kxf7 7. bxc3 Qd6 8. d4 B@g4 9. N@e5+ Nxe5 10. Nxe5+ Qxe5+ 11. dxe5 Bxd1 12. N@g5+ Kg6 13. Q@f7+ Kh6 14. Ne6+ P@d2+ 15. Bxd2+ { Black resigns. } 1-0

[bob]

Steven and I had many discussions about PGN over the years. His required 7 tags was OK back "when". Neither he nor anybody else ever envisioned what has happened today with capturing and encoding far more information than ever imagined back then.

My main complaint back then was with the EP field in the FEN standard he developed. Specifically that the EP square in the FEN string always shows the square behind a pawn that just advanced two squares, even if there was no enemy pawn on either adjacent file that could capture that pawn en passant. This was a little messy to deal with when a program saves a FEN position, as we don't consider an EP flag unless an EP capture can actually be made.

As far as PGN goes, it has been taken to the point of absurdity. I once had a PGN file that would break almost every engine at the time, mine included until I modified the code. Had lots of deep nested () and {} stuff that was really difficult to parse. Took a stack based parser to handle it. PGN was supposed to be readable by both humans and computers. But as ChessBase started to use these nested commands and suggestions, it became much harder for me to read. Fortunately Crafty could read 'em just fine and strip out the nested stuff. Then I could do a savepgn command and save a clean PGN file with just the actual game moves so that I could actually read and understand the move sequence...

Modifying it is always possible. But it needs to be done carefully so that it doesn't have to be then modified again, leading to a half-dozen different versions of the standard.

[sovaz1997]

Thank you for the suggestions!
Initially, the question was about creating a TCEC game viewer with their evals and other info for being able to view offline. But, since TCEC is not the only one to show ecals, a uniform standard is needed.

[lucasart]

Due to the fact that we have a huge number of different chess shells, I believe that it is necessary to create a new format for PGN files, which will standardize computer position evals, its analysis and additional data. With a single format, it will be much more convenient to use one PGN file in various chess shells.

For example we have TCEC comments format:

Code: Select all

{d=16, sd=51, mt=161000, tl=7049000, s=219832, n=35024512, pv=Nb3 Be7 h4 Rb8 h5 b5 Bd3 f6 exf6 Nxf6 Nd4 Na5 Qe2 Bb4 Kb1 Rb7 Nb3 Nc4 Bd4 Bd6 h6 g6 g3 Rff7 Bxc4, tb=0, h=86.3, ph=0.0, wv=1.08, R50=49, Rd=-11, Rr=-11, mb=+0+0+0+0+0,}

And tags:

Code: Select all

[Event "TCEC Season 16 - Superfinal"]
[Site "https://tcec-chess.com"]
[Date "2019.10.02"]
[Round "39.1"]
[White "AllieStein v0.5-dev_7b41f8c-n11"]
[Black "Stockfish 19092522"]
[Result "1-0"]
[BlackElo "3892"]
[ECO "C11"]
[GameDuration "03:50:43"]
[GameEndTime "2019-10-02T22:50:54.482 W. Europe Standard Time"]
[GameStartTime "2019-10-02T19:00:10.807 W. Europe Standard Time"]
[Opening "French"]
[PlyCount "120"]
[Termination "adjudication"]
[TerminationDetails "TCEC win rule"]
[TimeControl "7200+10"]
[Variation "Steinitz, Boleslavsky variation"]
[WhiteElo "3823"]

This is not standard. Different shells even in tags use different types of time recording (7200 + 10, 7200s + 10s).

Octagon GUI:

Code: Select all

[OS "Win10 x64"]
[CPU "i9-7960X CPU 32 Threads"]
[GPU "2 x GeForce RTX 2070"]
[Memory "64Gb"]
[TournamentFormat "Gauntlet"]
[WhiteTimeControl "7200s+10s"]
[BlackTimeControl "7200s+10s"]
[TimeOdds "1"]
[OptionsWhite ""]
[OptionsBlack ""]
[OpeningOptions ""]
[Tablebases ""]
[WinRule "1 moves, evals >=10.0 for 5 moves"]
[DrawRule "35 moves, evals <= 0.08 for 5 moves"]
[PlyCount "232"]
[Time "04:40:14433"]
[Termination "normal"]
[End Reason "White wins by adjudication rule"]

I suggest discussing on this topic: which tags really need to be standardized, which ones do not. In any case, this will be an extension of the PGN format, and not its replacement. I plan to write a standard, develop translators from different formats in PGN2 and write a test GUI that will show files of a new type.

The PGN format is an abortion…

But your proposals fail to adress the real problem of PGN which is its (un)parsability, and its use of SAN (another abortion), which requires to code full chess rules and board state to just parse moves.

In any case, you will not change anything by posting in an internet forum. Ever heard the phrase "he who codes decides" ? That's how open source progresses. Code talks, talk walks.

So if you want to move the needle *you* need to do the work of writing the code to parse you new format, and writing converter tools that are easy to use to convert to and from PGN. Even then, people won't use it because they won't get any practical benefit from it. There should be useful tools (eg. GUI) that *directly* input and out from/to this format. And *you* will have to do the work here, again.

[Ovyron]

Mandatory xkcd comic:

[hgm]

WinBoard/XBoard (and also Arena, I believe) use the convention that in a comment starting with <signed floating-point number>/<integer-number> these numbers indicate the score and search depth used by an engine to obtain the move before the comment. It can be optionally followed by a space and time indicator (encoding the time in seconds or <min>:<sec>) then tells how long the engine thought about that move.

This allows a simple and compact notation, which does not destroy readablity. WinBoard's implemetation does violate the origiall PGN standard in a minor way, though: this standard requires black moves after a comment to start with a move number plus three periods. This is a sensible requirement for long comments, but rather detrimental to readability in the case of the short score/depth comments. Therefore Winboard omits these black move numbers after comments that only contain the score/depth+time info.

Code: Select all

[Event "Computer Chess Game"]
[Site "ONTWIKKELLAPTOP"]
[Date "2019.10.07"]
[Round "-"]
[White "Fairy-Max 5.0b3"]
[Black "Fruit 2.1"]
[Result "0-1"]
[TimeControl "60"]

1. Nc3 {+0.28/7} d5 {-0.20/11 2.6} 2. Nf3 {+0.08/6 0.8} Nf6 {-0.23/10 1.4}
3. d4 {+0.02/6 1.7} Nc6 {-0.08/10 2.4} 4. Bf4 {+0.02/6 1.8} Nh5
{-0.21/10 2.3} 5. Qd2 {-0.02/5 0.6} Bf5 {+0.16/9 1.7} 6. Nb5 {+0.10/6 2.1}
Rc8 {+0.36/10 0.2} 7. O-O-O {+0.13/5 0.6} e6 {+0.83/9 1.7} 8. Be3
{-0.11/4 0.6} Be7 {+0.92/9 1.9} 9. Ne5 {-0.09/6 1.3} Nxe5 {+0.64/9 2.1} 10.
dxe5 {-0.13/6 1.0} Qd7 {+0.07/10 7} 11. Nxa7 {+0.41/6 1.4} Ra8
{+0.68/9 1.3} 12. Qc3 {+0.21/6 2.6} c5 {+1.18/10 0.1} 13. g4 {+0.17/6 0.9}
Be4 {+0.58/10 2.6} 14. f3 {+0.81/7 0.8} d4 {+0.23/9 0.3} 15. Qa3
{+0.74/6 0.6} Bd5 {+1.06/9 1.0} 16. c4 {+0.65/6 0.5} Bxc4 {+1.93/9 1.1} 17.
gxh5 {-0.09/6 0.6} b6 {+2.34/10 1.2} 18. b3 {-0.26/7 0.8} Rxa7
{+1.72/10 0.1} 19. Qb2 {-0.22/7 0.5} Bd5 {+1.87/9 0.6} 20. Bf4
{-0.23/6 0.5} O-O {+1.54/9 1.4} 21. h6 {+0.22/6 0.9} c4 {+1.61/8 1.1} 22.
Qxd4 {+0.77/5 1.2} cxb3 {+2.62/7 0.2} 23. axb3 {+0.41/5 0.8} Rc8+
{+3.94/7 0.3} 24. Kb1 {+0.19/5 1.1} Qb5 {+5.98/8 0.1} 25. Qd3 {+0.20/5 1.0}
Qa5 {+99.95/38 0.7} 26. Qa6 {-1000.02/19 0.4} Qxa6 {+99.97/35 0.2} 27. e4
{-1000.01/28 0.3} Qa1# {+99.99/35 0.6}
{Xboard adjudication: Checkmate} 0-1

I also want to point out that annotating a game with engine PVs hidden in a comment (as some GUIs do) is a very bad idea. The original standard provides 'recursive variations' (enclosed in parentheses) to encode alterative move sequences, and many GUIs (including WinBoard) have mechanisms that allow the user to view these move sequences with the same ease you can step through the main game. Hiding such a PV in a normal comment (within braces) hides these from the normal display mechanisms, and thusshould be considered a very bad practice.

[niklasf]

Hiding such a PV in a normal comment (within braces) hides these from the normal display mechanisms, and thusshould be considered a very bad practice.

Unfortunately the PV often coincides with the following moves of the game, and many GUIs don't support having distinct variations for the same moves. (And for a good reason, since most of the time that's not what users want.)