PGN standard, its improvement and standardization

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
sovaz1997
Posts: 235
Joined: Sun Nov 13, 2016 9:37 am

PGN standard, its improvement and standardization

Post by sovaz1997 » Sun Oct 06, 2019 12:47 pm

Due to the fact that we have a huge number of different chess shells, I believe that it is necessary to create a new format for PGN files, which will standardize computer position evals, its analysis and additional data. With a single format, it will be much more convenient to use one PGN file in various chess shells.

For example we have TCEC comments format:

Code: Select all

{d=16, sd=51, mt=161000, tl=7049000, s=219832, n=35024512, pv=Nb3 Be7 h4 Rb8 h5 b5 Bd3 f6 exf6 Nxf6 Nd4 Na5 Qe2 Bb4 Kb1 Rb7 Nb3 Nc4 Bd4 Bd6 h6 g6 g3 Rff7 Bxc4, tb=0, h=86.3, ph=0.0, wv=1.08, R50=49, Rd=-11, Rr=-11, mb=+0+0+0+0+0,}
And tags:

Code: Select all

[Event "TCEC Season 16 - Superfinal"]
[Site "https://tcec-chess.com"]
[Date "2019.10.02"]
[Round "39.1"]
[White "AllieStein v0.5-dev_7b41f8c-n11"]
[Black "Stockfish 19092522"]
[Result "1-0"]
[BlackElo "3892"]
[ECO "C11"]
[GameDuration "03:50:43"]
[GameEndTime "2019-10-02T22:50:54.482 W. Europe Standard Time"]
[GameStartTime "2019-10-02T19:00:10.807 W. Europe Standard Time"]
[Opening "French"]
[PlyCount "120"]
[Termination "adjudication"]
[TerminationDetails "TCEC win rule"]
[TimeControl "7200+10"]
[Variation "Steinitz, Boleslavsky variation"]
[WhiteElo "3823"]
This is not standard. Different shells even in tags use different types of time recording (7200 + 10, 7200s + 10s).

Octagon GUI:

Code: Select all

[OS "Win10 x64"]
[CPU "i9-7960X CPU 32 Threads"]
[GPU "2 x GeForce RTX 2070"]
[Memory "64Gb"]
[TournamentFormat "Gauntlet"]
[WhiteTimeControl "7200s+10s"]
[BlackTimeControl "7200s+10s"]
[TimeOdds "1"]
[OptionsWhite ""]
[OptionsBlack ""]
[OpeningOptions ""]
[Tablebases ""]
[WinRule "1 moves, evals >=10.0 for 5 moves"]
[DrawRule "35 moves, evals <= 0.08 for 5 moves"]
[PlyCount "232"]
[Time "04:40:14433"]
[Termination "normal"]
[End Reason "White wins by adjudication rule"]
I suggest discussing on this topic: which tags really need to be standardized, which ones do not. In any case, this will be an extension of the PGN format, and not its replacement. I plan to write a standard, develop translators from different formats in PGN2 and write a test GUI that will show files of a new type.
Zevra 2 - my chess engine.
Binary, source and description here: https://github.com/sovaz1997/Zevra2

BeyondCritics
Posts: 353
Joined: Sat May 05, 2012 12:48 pm
Location: Bergheim

Re: PGN standard, its improvement and standardization

Post by BeyondCritics » Sun Oct 06, 2019 6:18 pm

Principally i endorse your brave efforsts, but it won't work. The most you can hope for, is dying a peacuful dead, without anybody noticing.
Why? Because there are a lots of commercial and private vendors who do rely on the pgn standard and if you are succesful in changing that standard -- even in a fully "backward compatible" manner -- you will cost them money. You will cost them money and that makes them very nervous.
If you want to have at least a slim chance to change something to the better, you should address that issue upfront. This is only my opinion. Good luck anyway.

User avatar
hgm
Posts: 23772
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: PGN standard, its improvement and standardization

Post by hgm » Sun Oct 06, 2019 6:43 pm

Make sure to include the following tags:

* Variant
possible values e.g. suicide, atomic, capablanca, omega, xiangi, shogi, ...

* VariantMen
value is a comma-separated list of <piece id> : <Betza move descriptor> pairs for all unorthodox pieces participating in the game

Vinvin
Posts: 4390
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: PGN standard, its improvement and standardization

Post by Vinvin » Sun Oct 06, 2019 9:32 pm

Some tags from Lichess :

[Event "Rated Crazyhouse game"]
[Site "https://lichess.org/0E9HoiTK"]
[Date "2019.09.14"]
[Round "-"]
[White "chickencrossroad"]
[Black "gsvc"]
[Result "1-0"]
[UTCDate "2019.09.14"]
[UTCTime "00:34:12"]
[WhiteElo "2635"]
[BlackElo "2638"]
[WhiteRatingDiff "+7"]
[BlackRatingDiff "-8"]
[WhiteTitle "FM"]
[BlackTitle "IM"]
[Variant "Crazyhouse"]
[TimeControl "60+0"]
[ECO "?"]
[Opening "?"]
[Termination "Normal"]
[Annotator "lichess.org"]

1. e4 Nf6 2. Nc3 Nc6 3. Nf3 d5 4. exd5 Nxd5 5. Bc4 Nxc3 6. Bxf7+ Kxf7 7. bxc3 Qd6 8. d4 B@g4 9. N@e5+ Nxe5 10. Nxe5+ Qxe5+ 11. dxe5 Bxd1 12. N@g5+ Kg6 13. Q@f7+ Kh6 14. Ne6+ P@d2+ 15. Bxd2+ { Black resigns. } 1-0

bob
Posts: 20639
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: PGN standard, its improvement and standardization

Post by bob » Mon Oct 07, 2019 12:16 am

Steven and I had many discussions about PGN over the years. His required 7 tags was OK back "when". Neither he nor anybody else ever envisioned what has happened today with capturing and encoding far more information than ever imagined back then.

My main complaint back then was with the EP field in the FEN standard he developed. Specifically that the EP square in the FEN string always shows the square behind a pawn that just advanced two squares, even if there was no enemy pawn on either adjacent file that could capture that pawn en passant. This was a little messy to deal with when a program saves a FEN position, as we don't consider an EP flag unless an EP capture can actually be made.

As far as PGN goes, it has been taken to the point of absurdity. I once had a PGN file that would break almost every engine at the time, mine included until I modified the code. Had lots of deep nested () and {} stuff that was really difficult to parse. Took a stack based parser to handle it. PGN was supposed to be readable by both humans and computers. But as ChessBase started to use these nested commands and suggestions, it became much harder for me to read. Fortunately Crafty could read 'em just fine and strip out the nested stuff. Then I could do a savepgn command and save a clean PGN file with just the actual game moves so that I could actually read and understand the move sequence...

Modifying it is always possible. But it needs to be done carefully so that it doesn't have to be then modified again, leading to a half-dozen different versions of the standard.

sovaz1997
Posts: 235
Joined: Sun Nov 13, 2016 9:37 am

Re: PGN standard, its improvement and standardization

Post by sovaz1997 » Mon Oct 07, 2019 4:31 am

Thank you for the suggestions!
Initially, the question was about creating a TCEC game viewer with their evals and other info for being able to view offline. But, since TCEC is not the only one to show ecals, a uniform standard is needed.
Zevra 2 - my chess engine.
Binary, source and description here: https://github.com/sovaz1997/Zevra2

User avatar
lucasart
Posts: 3041
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: PGN standard, its improvement and standardization

Post by lucasart » Mon Oct 07, 2019 4:45 am

sovaz1997 wrote:
Sun Oct 06, 2019 12:47 pm
Due to the fact that we have a huge number of different chess shells, I believe that it is necessary to create a new format for PGN files, which will standardize computer position evals, its analysis and additional data. With a single format, it will be much more convenient to use one PGN file in various chess shells.

For example we have TCEC comments format:

Code: Select all

{d=16, sd=51, mt=161000, tl=7049000, s=219832, n=35024512, pv=Nb3 Be7 h4 Rb8 h5 b5 Bd3 f6 exf6 Nxf6 Nd4 Na5 Qe2 Bb4 Kb1 Rb7 Nb3 Nc4 Bd4 Bd6 h6 g6 g3 Rff7 Bxc4, tb=0, h=86.3, ph=0.0, wv=1.08, R50=49, Rd=-11, Rr=-11, mb=+0+0+0+0+0,}
And tags:

Code: Select all

[Event "TCEC Season 16 - Superfinal"]
[Site "https://tcec-chess.com"]
[Date "2019.10.02"]
[Round "39.1"]
[White "AllieStein v0.5-dev_7b41f8c-n11"]
[Black "Stockfish 19092522"]
[Result "1-0"]
[BlackElo "3892"]
[ECO "C11"]
[GameDuration "03:50:43"]
[GameEndTime "2019-10-02T22:50:54.482 W. Europe Standard Time"]
[GameStartTime "2019-10-02T19:00:10.807 W. Europe Standard Time"]
[Opening "French"]
[PlyCount "120"]
[Termination "adjudication"]
[TerminationDetails "TCEC win rule"]
[TimeControl "7200+10"]
[Variation "Steinitz, Boleslavsky variation"]
[WhiteElo "3823"]
This is not standard. Different shells even in tags use different types of time recording (7200 + 10, 7200s + 10s).

Octagon GUI:

Code: Select all

[OS "Win10 x64"]
[CPU "i9-7960X CPU 32 Threads"]
[GPU "2 x GeForce RTX 2070"]
[Memory "64Gb"]
[TournamentFormat "Gauntlet"]
[WhiteTimeControl "7200s+10s"]
[BlackTimeControl "7200s+10s"]
[TimeOdds "1"]
[OptionsWhite ""]
[OptionsBlack ""]
[OpeningOptions ""]
[Tablebases ""]
[WinRule "1 moves, evals >=10.0 for 5 moves"]
[DrawRule "35 moves, evals <= 0.08 for 5 moves"]
[PlyCount "232"]
[Time "04:40:14433"]
[Termination "normal"]
[End Reason "White wins by adjudication rule"]
I suggest discussing on this topic: which tags really need to be standardized, which ones do not. In any case, this will be an extension of the PGN format, and not its replacement. I plan to write a standard, develop translators from different formats in PGN2 and write a test GUI that will show files of a new type.
The PGN format is an abortion…

But your proposals fail to adress the real problem of PGN which is its (un)parsability, and its use of SAN (another abortion), which requires to code full chess rules and board state to just parse moves.

In any case, you will not change anything by posting in an internet forum. Ever heard the phrase "he who codes decides" ? That's how open source progresses. Code talks, talk walks.

So if you want to move the needle *you* need to do the work of writing the code to parse you new format, and writing converter tools that are easy to use to convert to and from PGN. Even then, people won't use it because they won't get any practical benefit from it. There should be useful tools (eg. GUI) that *directly* input and out from/to this format. And *you* will have to do the work here, again.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

User avatar
Ovyron
Posts: 2818
Joined: Tue Jul 03, 2007 2:30 am

Re: PGN standard, its improvement and standardization

Post by Ovyron » Mon Oct 07, 2019 4:57 am

Mandatory xkcd comic:

Image
Great spirits have always encountered violent opposition from mediocre minds.

User avatar
hgm
Posts: 23772
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: PGN standard, its improvement and standardization

Post by hgm » Mon Oct 07, 2019 7:39 am

WinBoard/XBoard (and also Arena, I believe) use the convention that in a comment starting with <signed floating-point number>/<integer-number> these numbers indicate the score and search depth used by an engine to obtain the move before the comment. It can be optionally followed by a space and time indicator (encoding the time in seconds or <min>:<sec>) then tells how long the engine thought about that move.

This allows a simple and compact notation, which does not destroy readablity. WinBoard's implemetation does violate the origiall PGN standard in a minor way, though: this standard requires black moves after a comment to start with a move number plus three periods. This is a sensible requirement for long comments, but rather detrimental to readability in the case of the short score/depth comments. Therefore Winboard omits these black move numbers after comments that only contain the score/depth+time info.

Code: Select all

[Event "Computer Chess Game"]
[Site "ONTWIKKELLAPTOP"]
[Date "2019.10.07"]
[Round "-"]
[White "Fairy-Max 5.0b3"]
[Black "Fruit 2.1"]
[Result "0-1"]
[TimeControl "60"]

1. Nc3 {+0.28/7} d5 {-0.20/11 2.6} 2. Nf3 {+0.08/6 0.8} Nf6 {-0.23/10 1.4}
3. d4 {+0.02/6 1.7} Nc6 {-0.08/10 2.4} 4. Bf4 {+0.02/6 1.8} Nh5
{-0.21/10 2.3} 5. Qd2 {-0.02/5 0.6} Bf5 {+0.16/9 1.7} 6. Nb5 {+0.10/6 2.1}
Rc8 {+0.36/10 0.2} 7. O-O-O {+0.13/5 0.6} e6 {+0.83/9 1.7} 8. Be3
{-0.11/4 0.6} Be7 {+0.92/9 1.9} 9. Ne5 {-0.09/6 1.3} Nxe5 {+0.64/9 2.1} 10.
dxe5 {-0.13/6 1.0} Qd7 {+0.07/10 7} 11. Nxa7 {+0.41/6 1.4} Ra8
{+0.68/9 1.3} 12. Qc3 {+0.21/6 2.6} c5 {+1.18/10 0.1} 13. g4 {+0.17/6 0.9}
Be4 {+0.58/10 2.6} 14. f3 {+0.81/7 0.8} d4 {+0.23/9 0.3} 15. Qa3
{+0.74/6 0.6} Bd5 {+1.06/9 1.0} 16. c4 {+0.65/6 0.5} Bxc4 {+1.93/9 1.1} 17.
gxh5 {-0.09/6 0.6} b6 {+2.34/10 1.2} 18. b3 {-0.26/7 0.8} Rxa7
{+1.72/10 0.1} 19. Qb2 {-0.22/7 0.5} Bd5 {+1.87/9 0.6} 20. Bf4
{-0.23/6 0.5} O-O {+1.54/9 1.4} 21. h6 {+0.22/6 0.9} c4 {+1.61/8 1.1} 22.
Qxd4 {+0.77/5 1.2} cxb3 {+2.62/7 0.2} 23. axb3 {+0.41/5 0.8} Rc8+
{+3.94/7 0.3} 24. Kb1 {+0.19/5 1.1} Qb5 {+5.98/8 0.1} 25. Qd3 {+0.20/5 1.0}
Qa5 {+99.95/38 0.7} 26. Qa6 {-1000.02/19 0.4} Qxa6 {+99.97/35 0.2} 27. e4
{-1000.01/28 0.3} Qa1# {+99.99/35 0.6}
{Xboard adjudication: Checkmate} 0-1
I also want to point out that annotating a game with engine PVs hidden in a comment (as some GUIs do) is a very bad idea. The original standard provides 'recursive variations' (enclosed in parentheses) to encode alterative move sequences, and many GUIs (including WinBoard) have mechanisms that allow the user to view these move sequences with the same ease you can step through the main game. Hiding such a PV in a normal comment (within braces) hides these from the normal display mechanisms, and thusshould be considered a very bad practice.

niklasf
Posts: 41
Joined: Sat May 16, 2015 9:41 pm

Re: PGN standard, its improvement and standardization

Post by niklasf » Mon Oct 07, 2019 9:16 am

hgm wrote:
Mon Oct 07, 2019 7:39 am
Hiding such a PV in a normal comment (within braces) hides these from the normal display mechanisms, and thusshould be considered a very bad practice.
Unfortunately the PV often coincides with the following moves of the game, and many GUIs don't support having distinct variations for the same moves. (And for a good reason, since most of the time that's not what users want.)

Post Reply