FRC in CECP

[Ras]

It's actually even worse because the ecosystem is pretty fragmented. With CECP engines, you can't rely on anything in practice, half of the features you'd expect don't work, and what works does so in whatever way. That's my experience with CECP engines in general, and why I stopped even using them for testing. Too much hassle.

Let's have a look at Crafty. Given that Bob has always been very much in favour of CECP, and Crafty has been around for a long time, you would expect that things work nicely. Crafty is CECP v2 btw, and doesn't support "usermove".

Crafty requests SAN output in engine answers through "san=1" by the engine at startup. This is one of the very few opportunities where the otherwise pretty horrible two way feature negotiation might make at least some sense, to let's reject this "feature". Of course, Crafty ignores the rejection...

Code: Select all

ERROR.  feature san rejected by xboard

... and still prints out everything in SAN.

Here's what it looks like with the full game move numbering and the omission dots when Black is to move:

Code: Select all

10    416       7  66408  4. ... d4 5. Nb5 Nc6 6. e3 e5 7. exd4 exd4 8. Bc4 Be6 9. Bxe6 fxe6

Why isn't that an issue? Because the PV is just some opaque text anyway, intended to be displayed to the user as is. You could even put in some poem. Obviously, this makes it hard to implement some basic mouse-over feature that would display the board at each point of the PV as a pop-up tool tip, or that would work only with some engine-GUI pairings.

Then the CECP spec has this: "SCORE is in centi-Pawn units, and the de-facto standard is to report it from the point of view of the side to move." A spec has one job, to define and mandate a standard, and it can't even define the score sign.

Can you guess what Crafty does? Yup, it always prints White's POV for the score. The 416 in the output above is because White is a knight up and Black is to move.

That's usually "solved" in the GUI by forcing the user to tick some config box. If you think about automating that in c-chess-cli for adjudication, there is no box, and the best way is probably to auto-test a position that has a clear advantage for White (like being a rook up), setting Black to move, and see what the score sign is.

You can't even assume that a CECP engine can play more than one game - some will just exit at the end of a game, and you have to start the executable again. So you need to check whether the engine announces "reuse=1" at startup.

My take is that if you want to fully support CECP engines in c-chess-cli, that will be a whole lot of work with unreliable operation in practice.

[hgm]

Note that the san feature only applies to GUI -> engine traffic. The specs say that SAN is always allowed for engine->GUI. So even though Crafty is in general horribly non-compliant, it is not at fault on this count.

Of course allowing SAN in either direction was a terribly bad idea; no need to stress that point. In practice it is not really a pain for the GUI developer, though, as GUIs typically have to deal with SAN for reading / writing PGN anyway. So they do tend to have SAN parsers already, and it is just a matter of invoking them. In XBoard I just feed the engine's PV to the PGN parser to get the move sequence, and rid it of move numbers and other frivolities.

[lucasart]

But sub-second timing is never recommended when not running on a bare machine; scheduling algorithms of operating systems would introduce a lot of noise, degrading the accuracy of the result.

Right, so sub-second is not possible because of the defective operating system. Not because of the defective protocol

[lucasart]

It's actually even worse because the ecosystem is pretty fragmented. With CECP engines, you can't rely on anything in practice, half of the features you'd expect don't work, and what works does so in whatever way. That's my experience with CECP engines in general, and why I stopped even using them for testing. Too much hassle.

Let's have a look at Crafty. Given that Bob has always been very much in favour of CECP, and Crafty has been around for a long time, you would expect that things work nicely. Crafty is CECP v2 btw, and doesn't support "usermove".

Crafty requests SAN output in engine answers through "san=1" by the engine at startup. This is one of the very few opportunities where the otherwise pretty horrible two way feature negotiation might make at least some sense, to let's reject this "feature". Of course, Crafty ignores the rejection...

Code: Select all

ERROR.  feature san rejected by xboard

... and still prints out everything in SAN.

Here's what it looks like with the full game move numbering and the omission dots when Black is to move:

Code: Select all

10    416       7  66408  4. ... d4 5. Nb5 Nc6 6. e3 e5 7. exd4 exd4 8. Bc4 Be6 9. Bxe6 fxe6

Why isn't that an issue? Because the PV is just some opaque text anyway, intended to be displayed to the user as is. You could even put in some poem. Obviously, this makes it hard to implement some basic mouse-over feature that would display the board at each point of the PV as a pop-up tool tip, or that would work only with some engine-GUI pairings.

Then the CECP spec has this: "SCORE is in centi-Pawn units, and the de-facto standard is to report it from the point of view of the side to move." A spec has one job, to define and mandate a standard, and it can't even define the score sign.

Can you guess what Crafty does? Yup, it always prints White's POV for the score. The 416 in the output above is because White is a knight up and Black is to move.

That's usually "solved" in the GUI by forcing the user to tick some config box. If you think about automating that in c-chess-cli for adjudication, there is no box, and the best way is probably to auto-test a position that has a clear advantage for White (like being a rook up), setting Black to move, and see what the score sign is.

You can't even assume that a CECP engine can play more than one game - some will just exit at the end of a game, and you have to start the executable again. So you need to check whether the engine announces "reuse=1" at startup.

My take is that if you want to fully support CECP engines in c-chess-cli, that will be a whole lot of work with unreliable operation in practice.

Crafty is an interesting example. It is the ONLY engine that is worthy of interest, which does not support UCI, and therefore requires CECP support. Otherwise all engines support UCI, even the old ones that used to be CECP. So if Crafty's CECP implementation is broken beyond repair, and given how stubborn Bob is (he won't fix it, because he won't admit it's wrong in the first place), it's a waste of time to support CECP. Still, it was interesting to investigate this protocol, to see how deep the rabbit hole can go.

I think we could easily a 10 page pamphlet "CECP considered harmful", and we would barely scratch the surface. Mocking CECP is like shooting fish in a barrel

[hgm]

Right, so sub-second is not possible because of the defective operating system. Not because of the defective protocol

Exactly. Shortcomings in a protocol can be cured with the stroke of a pen. For real-life issues this is not so easy. It makes no sense to build commands into a protocol for ordering things that would never work in practice, or are bound to work poorly. It would have been very easy to define a CECP feature microsec=1, which would request specifying the time in units of microsec in the 'level' and 'time'/ 'otim' commands. But I would consider that harmful. Because it suggests to the user the engine can do something which it cannot possibly do in real life.

To do reliable testing at such speeds, TIN is the only way. So CECP supports TIN (feature nps=1) rather than sub-centi-second timing (feature microsec=1). Note, btw, that in practice you always have centi-sec timing precision, because the base time in the 'level' command is immediately overruled by the actual start setting of the clock, transferred by 'time' in centi-sec units. The base time is really only relevant in classical TCs, to give the engine a preview of how much time it will get added to its clock when the current session expires. Something that cannot be told to the engine at all in UCI, I might add...

[Guenther]

...

My take is that if you want to fully support CECP engines in c-chess-cli, that will be a whole lot of work with unreliable operation in practice.

Crafty is an interesting example. It is the ONLY engine that is worthy of interest, which does not support UCI, and therefore requires CECP support. Otherwise all engines support UCI, even the old ones that used to be CECP.
...

As this thread changed since long from 'FRC in CECP' to plain CECP, I have to say above quote is bullshit.

Of course you still can use any engine with WB2UCI, but above statement is just uninformed bashing...
If only Crafty is interesting for you it may be so (for unknown reason), but it is not for other people

Code: Select all

(CCRL or GURL single core)
Rating:         Class
> 3100          A1
2900-3100       A2
2700-2900       A3
2500-2700       B1
2300-2500       B2

XB only list - only the engines with at least 'B2' strength (disregarded a few hundreds weaker ones)
(note: to my best knowledge - please report errors for correction in the XB/UCI chronology, if you find some)

Code: Select all

Name            Last Release    RC      Src     PL      Prot
Amateur         2016-10         B2      -       C       XB
Amundsen        2009-04         B2      yes     C       XB
Amy             2004-03         B2      yes     C       XB
Anatoli         2008-04         B1      -       Pascal  XB
Ant (& Gi-Ant)  2010-12         B2      -       C++     XB
Averno          2005-03         B2      -       C++     XB
Bearded         2012-03         B2      -               XB
Betsy           2003-04         B2      -       C       XB
Bringer (Der)   2001-12         B2      -       C++     XB
Bruja           2010-05         B2      -       C++     XB
BugChess2       2011-04         A3      -       C++     XB
Butcher         2012-04         B2      -       C       XB
Caligula#       2019-03         B1      -               XB
Chezzz          2003-02         B2      -       C++     XB
Chronos         2010-06         A3      -       C++     XB
Comet           2005-05         B2      -       C       XB
Crafty          2020-02         A2      yes     C       XB
Cyberpagno      2016-06         B2      -       C       XB
DanChess        2005-02         B1      -       C++     XB
Darmenios       2010-10         B2      -       C++     XB
Dimitri         2019-10         B1*     -               XB
Dirty           2018-11         A2*     -       C++     XB
Djinn           2014-04         A3      -       C++     XB
Dorky           2018-08         B2      -       C++     XB
Drosophila      2020-03         B2*     yes     C       XB
Eeyore          2010-11         B2      -               XB
ETChess         2008-01         B1      -               XB
EveAnn          2017-04         B2      -       Pascal  XB
EXchess         2017-02         A2      yes     C++     XB
Francesca       2020-06         A3*     -       C       XB
FrankWalter     2019-05         B2      yes     Java    XB
Ghost           2017-05         B2      -       C++     XB
Giraffe         2016-11         B2*     yes     C++     XB
Gosu            2006-10         B2      -               XB
GreenLightChess 2004-03         B1      -       C++     XB
Gromit3         2002-01         B2      -       C++     XB
Horizon         2008-02         B2      -       C       XB
JikChess        2015-10         B1      -       C++     XB
Joker           2007-12         B2      -       C       XB
Jumbo           2019-07         B1*     -       C++     XB
KingOfKings     2004-09         B2      -       C++     XB
Kiwi            2005-11         B2      yes     C++     XB
KnightDreamer   2004-07         B2      -               XB
Matacz          2008-09         B2      -               XB
Movei           2007-08         B1      -       C       XB
Myrddin         2015-01         B2      -               XB
nanoSzachy      2015-10         B1      -               XB
Nebiyu          2020-01         B2*     yes     C++     XB
Nemeton         2019-05         B2*     yes     Pascal  XB
Nimzo           1999-11         B1      -               XB
NoraGrace       2014-11         B1      yes     C#      XB
Olithink        2020-12         A3      yes     C       XB
Petir           2007-09         B1      -               XB
Phalanx         2016-05         B1*     yes     C       XB
pikoSzachy      2015-10         B2      -               XB
Popochin#       2018-04         B2      -               XB
Porucnik        2009-01         B2      yes     C++     XB
Postmodernist   2006-06         B2      -               XB
ProDeo          2020-11         A3      -               XB
Prophet         2007-02         B2*     yes     C       XB
Prophet3        2019-06         B2*     yes     C++     XB
Pseudo          2005-04         B1      -       C       XB
Quark           2004-07         B2      -       C++     XB
RattateChess    2010-02         B2      -               XB
Resp            2010-12         B2      yes     C++     XB
Rodin           2016-02         B1      -       C       XB
RomiChess       2017-12         B2      -       C       XB
Sage            2008-05         B2      -               XB
Sloppy#         2010-07         B1      yes     C       XB
SpiderChess     2007-06         B2      -       Pascal  XB
TheCrazyBishop  2004-07         B2      -       C++     XB
TheKing         2007-10         B1      -       C       XB
Thinker*        2009-03         A2      -       C++     XB
Thor's Hammer   2010-10         B2      yes     C++     XB
TJChess         2016-09         B2      -               XB
Trace#          2006-11         B2      -       C       XB
Typhoon         2008-08         B2      yes     C       XB
Tytan           2007-07         B2      -       C++     XB
Waxman          2018-01         B2      -       C       XB
WChess          2018-05         B2      -       C       XB
Xpdnt           2009-10         B2      yes     C       XB
Zarkov          2016-06         B1      -       C       XB

[Ras]

But sub-second timing is never recommended when not running on a bare machine; scheduling algorithms of operating systems would introduce a lot of noise, degrading the accuracy of the result.

The fastest time mode my engine can reliably deal with is one second per game with no increments. That works even when I load my CPU with eight games in parallel, given that this CPU is a quadcore with SMT. Though c-chess-cli is stricter on time forfeits than Arena so that my upcoming release will improve the time allocation for such super-fast games.

Note that the san feature only applies to GUI -> engine traffic. The specs say that SAN is always allowed for engine->GUI. So even though Crafty is in general horribly non-compliant, it is not at fault on this count.

The spec is a bit unclear on that one and says "Some GUIs understand SAN. (But don't count on it!)" - which sounds more like a description of the status quo than like an actual requirement.

it's a waste of time to support CECP.

I understand this as in "CECP for c-chess-cli" - but given that commit number 321 has merged the PR for basic CECP support already, what's the way forward? Making it fully work? Removing it? Leaving it in half-working state?

[lucasart]

it's a waste of time to support CECP.

I understand this as in "CECP for c-chess-cli" - but given that commit number 321 has merged the PR for basic CECP support already, what's the way forward? Making it fully work? Removing it? Leaving it in half-working state?

Removing it.

It's not even close to half, actually. Below are some examples; enough to double the code size/complexity/fragility.

• new can't be used, because it's an opt-in feature. If engine doesn't have it, it must be stopped/restarted for each game.
• setboard is a feature, if engine doesn't have it, we must abort the tournament, or disqualify the engine and continue with the others.
• the only way for engines without setboard support, is to support opening formats other than FEN, where you start from the startpos, and record the list of moves. for example polyglot book format.
• resign doesn't work either, because of special snowflakes like Crafty. At UI level, you need a per-engine resign option, instead of a global one (so Crafty can opt out). Then, for Crafty, one has to edit an ini file to configure crafty's resign parameters, and add support resign requests in the UI.
• and what about the "offer draw" mechanism ? is this a joke, or am I seriously supposed to support this ?
• oh, it gets, better. the engine can send its game result determination the UI, like 1-0, 0-1, 1/2-1/2. then the UI will check if it's correct, and the engine loses for making a false determination. again, is this a joke ? maybe they forgot to implement the "j'adoube" rule for computers as well, where the engine would get a bitmap of the board, and see if the pieces are nicely centered.

CECP should have died 2 decades ago. Supporting it is a disservice to the community, because it only encourages more people to waste their time with it, instead of writing UCI engines (which is much easier btw).

[lucasart]

it's a waste of time to support CECP.

I understand this as in "CECP for c-chess-cli" - but given that commit number 321 has merged the PR for basic CECP support already, what's the way forward? Making it fully work? Removing it? Leaving it in half-working state?

Removing it.

It's not even close to half, actually. Below are some examples; enough to double the code size/complexity/fragility.

• new can't be used, because it's an opt-in feature. If engine doesn't have it, it must be stopped/restarted for each game.
• setboard is a feature, if engine doesn't have it, we must abort the tournament, or disqualify the engine and continue with the others.
• the only way for engines without setboard support, is to support opening formats other than FEN, where you start from the startpos, and record the list of moves. for example polyglot book format.
• resign doesn't work either, because of special snowflakes like Crafty. At UI level, you need a per-engine resign option, instead of a global one (so Crafty can opt out). Then, for Crafty, one has to edit an ini file to configure crafty's resign parameters, and add support resign requests in the UI.
• and what about the "offer draw" mechanism ? is this a joke, or am I seriously supposed to support this ?
• oh, it gets, better. the engine can send its game result determination the UI, like 1-0, 0-1, 1/2-1/2. then the UI will check if it's correct, and the engine loses for making a false determination. again, is this a joke ? maybe they forgot to implement the "j'adoube" rule for computers as well, where the engine would get a bitmap of the board, and see if the pieces are nicely centered.

CECP should have died 2 decades ago. Supporting it is a disservice to the community, because it only encourages more people to waste their time with it, instead of writing UCI engines (which is much easier btw).

Oh, I forgot. In c-chess-cli, I parse the PVs, to check legality (useful for uncovering bugs). With CECP, this is impossible, because there is no standard format. Not only SAN is allowed (for no valid technical reason whatsoever), and writing a SAN decoder is a huge pain. But, it's much worse. There is no specified format for PVs. They come totally free style, as illustrated by the Crafty example. An unparsable mess.

[Ras]

Of course you still can use any engine with WB2UCI

Unless I'm mistaken, that's Windows-only and closed source - i.e. useless in this case. Polyglot is OSS, but converts the other way around.

Removing it. It's not even close to half, actually.

OK, that's a clear way. I was already wondering when I saw the PR and didn't spot SAN handling, engine restart, or score sign auto-detection.

In c-chess-cli, I parse the PVs, to check legality (useful for uncovering bugs).

Indeed - it found a PV bug in my engine in the extremely rare case that PxQ with promotion is possible somewhere down the PV, but not the best move in that position.