what happened to scorpio NN in TCEC?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
tpoppins
Posts: 919
Joined: Tue Nov 24, 2015 8:11 pm
Location: upstate

Re: what happened to scorpio NN in TCEC?

Post by tpoppins » Sun Nov 18, 2018 12:34 am

Branko Radovanovic wrote:
Sat Nov 17, 2018 7:19 pm
How can one hurt fairness by including non-crashing games of a crashing engine?
Very simple, by introducing the so-called "tester bias". I'll illustrate from an example from my own practice.

Early in my testing career I was testing an old Winboard engine under Cute Chess GUI and observed an unusually high proportion (over 30%) of abnormal terminations, mostly stalled connections. My standard practice is to replay such games with exact same opening lines, and that I proceeded to do. The crashes persisted, albeit on progressively smaller scale, with some games having to be replayed three-four times; I, too, was bent on getting nice clean results and persevered.

When the test was complete at last I found to my astonishment that the engine performance exceeded its current rating by over 250 Elo! Why? No losses, and just a handful of draws, including results against opponents that were guaranteed to draw some of its "blood".

A closer look at the mechanics of the situation revealed the reason for the stalls: the engine was trying to resign in lost positions but did that in a nonstandard way, by sending "computer resigns" to the GUI instead of the proper "(computer) resign". The GUI ignored the nonstandard command while the engine ceased trying to communicate, considering the game over. And by replaying the games until the "crashes" disappeared I unintentionally introduced severe distortion into the results. In effect it amounted to manually picking the wins and draws and discarding all losses. Hence the term, "tester bias".

Just another example of how naively following the dictums of common sense while failing to consider hidden nuances can sometimes lead to unexpected and undesirable results.

Daniel Shawul wrote:
Sat Nov 17, 2018 5:41 pm
Edit:
Indeed cutechess implements things correctly like I suspected. It waits for a "feature done 1" before initializing.

Code: Select all

	else if (name == "done")
	{
		write("accepted done", Unbuffered);
		m_initTimer->stop();
		
		if (val == "1")
			initialize();
		return;
	}
That code may not be present in the old TCEC fork of cutechess if they are still using it. There were commits to the master branch related to hanging engines only a few months ago.
Tirsa Poppins
CCRL

AndrewGrant
Posts: 490
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: what happened to scorpio NN in TCEC?

Post by AndrewGrant » Sun Nov 18, 2018 5:35 am

Whats more likely...

1) There is an issue with cute-chess that seemingly only impacts Scorpio

2) An engine which has crashed out in all recent seasons (and denied entrance for one as a result), which has recently had a NN and MCTS component strapped to it, might just happen to crash because there is a problem with the engine.

Stability is #1 goal. Most authors know that.

User avatar
lucasart
Posts: 3041
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: what happened to scorpio NN in TCEC?

Post by lucasart » Sun Nov 18, 2018 6:15 am

CMCanavessi wrote:
Sat Nov 17, 2018 11:47 pm
lucasart wrote:
Sat Nov 17, 2018 11:40 pm
This tournament is becoming a bit of a mess:

They were supposed to restart without Scorpio, but they just restarted and forgot the "without Scorpio" part…
They did, but they aborted it after 3 games and re-started it with Scorpio in again, apparently with the same version they used for teh blitz test games (scorpio did 80 games without crashing even once)

What about Pirarucu? It had 3 crashes, and they said 3 crashes were disqualifying…
From what I've read, it crashed 2 times, the 3rd was a problem with something else

Again this will stall the live every time Pirarucu or Scorpio crashes (annoying), and will pollute results (unfair). Some engines will get lucky and "win" against Pirarucu due tocrashes, where they otherwise would have lost or drawn, some will not be gifted with opponent crashes.

As for Scorpio, it cannot pollute the results, because it would lose every single game regardless.
Let's hope now that things have stabilized no crashes will appear.
What has stabilized ? Nothing has been fixed. The same causes will have the same effect. Scorpio has already crashed in its 1st game. Pirarucu will crash too, unless it gets very lucky.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

Daniel Shawul
Posts: 3758
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: what happened to scorpio NN in TCEC?

Post by Daniel Shawul » Sun Nov 18, 2018 9:28 am

AndrewGrant wrote:
Sun Nov 18, 2018 5:35 am
Whats more likely...

1) There is an issue with cute-chess that seemingly only impacts Scorpio

2) An engine which has crashed out in all recent seasons (and denied entrance for one as a result), which has recently had a NN and MCTS component strapped to it, might just happen to crash because there is a problem with the engine.

Stability is #1 goal. Most authors know that.
It seems you feel threatened with a different approach and constantly attacks scorpio with "crashes" it has become annoying.
You have no idea if it is the cause of it too.
Better you stick with your well tested single algorithm and tuning it with a gigantic distirbuted project to death, but that is not my cup of tea.

1) I do not have cutechess-scorpio issues here for many games. Infact the same executable played HGM's tourney today 10 games no problems with xboard.

2) What that has got to do with anything ? It was playing with alpha-beta search + ABDADA in the last version ?
You know it could have just used alpha-beta + YBW and it wouldn't have crashed since that has been tested for years.

Daniel

AndrewGrant
Posts: 490
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: what happened to scorpio NN in TCEC?

Post by AndrewGrant » Sun Nov 18, 2018 10:04 am

Daniel Shawul wrote:
Sun Nov 18, 2018 9:28 am
[...] constantly attacks scorpio with "crashes" [...]
You've crashed out for every single season you have ever took place in.

Maybe its time to fix some things?

Daniel Shawul
Posts: 3758
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: what happened to scorpio NN in TCEC?

Post by Daniel Shawul » Sun Nov 18, 2018 10:12 am

AndrewGrant wrote:
Sun Nov 18, 2018 10:04 am
Daniel Shawul wrote:
Sun Nov 18, 2018 9:28 am
[...] constantly attacks scorpio with "crashes" [...]
You've crashed out for every single season you have ever took place in.

Maybe its time to fix some things?
Pathetic little troll. Scorpio has been playing before you were born ...

AndrewGrant
Posts: 490
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: what happened to scorpio NN in TCEC?

Post by AndrewGrant » Sun Nov 18, 2018 10:34 am

Daniel Shawul wrote:
Sun Nov 18, 2018 10:12 am
AndrewGrant wrote:
Sun Nov 18, 2018 10:04 am
Daniel Shawul wrote:
Sun Nov 18, 2018 9:28 am
[...] constantly attacks scorpio with "crashes" [...]
You've crashed out for every single season you have ever took place in.

Maybe its time to fix some things?
Pathetic little troll. Scorpio has been playing before you were born ...
Its not trolling. I legitimately think its shameful that you continue to submit Scoprio to TCEC, full well knowing it is going to crash and cause issues.

KnightMoves said he would never allow Scorpio back in after the failed "fixes" you claimed in a previous season. Its the one thing he and I were on the same page about.

Daniel Shawul
Posts: 3758
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: what happened to scorpio NN in TCEC?

Post by Daniel Shawul » Sun Nov 18, 2018 10:49 am

AndrewGrant wrote:
Sun Nov 18, 2018 10:34 am
Daniel Shawul wrote:
Sun Nov 18, 2018 10:12 am
AndrewGrant wrote:
Sun Nov 18, 2018 10:04 am
Daniel Shawul wrote:
Sun Nov 18, 2018 9:28 am
[...] constantly attacks scorpio with "crashes" [...]
You've crashed out for every single season you have ever took place in.

Maybe its time to fix some things?
Pathetic little troll. Scorpio has been playing before you were born ...
Its not trolling. I legitimately think its shameful that you continue to submit Scoprio to TCEC, full well knowing it is going to crash and cause issues.

KnightMoves said he would never allow Scorpio back in after the failed "fixes" you claimed in a previous season. Its the one thing he and I were on the same page about.
For a guy who complained and complained to get the previous admin out off TCEC to run away with his GUI, you should not be talking like that.
Like I said you better go back to your comfort zone and tune and tune ...

To all others who are interested in this issue, I have managed to reproduce the issue with cutechess-cli by inserting a fake wait for 2 minutes (though
NN loading takes 0 seconds to load).

Indeed cutechess exits (which it shouldn't have) since the engine didn't send a "feature done 1". Xboard has no problems with it. I palyed scorpio-scorpio after both took 2 minutes to load neural networks....

Clearly cutechess-cli doesn't implement the winboard protocol correctly despite what the code suggests..

Code: Select all

8 >scorpio-nn1(0): xboard
8 >scorpio-nn1(0): protover 2
9 >scorpio-nn2(1): xboard
9 >scorpio-nn2(1): protover 2
11 <scorpio-nn1(0): feature done=0
11 >scorpio-nn1(0): accepted done
12 <scorpio-nn2(1): feature done=0
12 >scorpio-nn2(1): accepted done
21 <scorpio-nn1(0): ht 4194304 X 16 = 64.0 MB
21 <scorpio-nn1(0): eht 524288 X 8 = 8.0 MB
21 <scorpio-nn1(0): pht 32768 X 24 = 0.8 MB
21 <scorpio-nn1(0): treeht 335539200 X 40 = 12799.8 MB
21 <scorpio-nn2(1): ht 4194304 X 16 = 64.0 MB
21 <scorpio-nn2(1): eht 524288 X 8 = 8.0 MB
21 <scorpio-nn2(1): pht 32768 X 24 = 0.8 MB
21 <scorpio-nn2(1): treeht 335539200 X 40 = 12799.8 MB
42 <scorpio-nn2(1): processors [16]
43 <scorpio-nn1(0): processors [16]
129 <scorpio-nn1(0): EgbbProbe 4.3 by Daniel Shawul
129 <scorpio-nn2(1): EgbbProbe 4.3 by Daniel Shawul
180 egbbs loaded !      ding egbbs....
376 <scorpio-nn1(0): Loading neural network : nets/net-0.pb
378 <scorpio-nn1(0): Loading graph on /cpu:0
180 egbbs loaded !      ding egbbs....
384 <scorpio-nn2(1): Loading neural network : nets/net-26.pb
386 <scorpio-nn2(1): Loading graph on /cpu:0
449 <scorpio-nn1(0): Neural network loaded !	
449 <scorpio-nn1(0): loading_time = 0s
462 <scorpio-nn2(1): Neural network loaded !	
462 <scorpio-nn2(1): loading_time = 0s
Engine scorpio-nn1(0) did not start the chess protocol in time
Terminating process of engine scorpio-nn1(0)
Engine scorpio-nn2(1) did not start the chess protocol in time
Terminating process of engine scorpio-nn2(1)
Finished game 1 (scorpio-nn1 vs scorpio-nn2): * {No result}
Score of scorpio-nn1 vs scorpio-nn2: 0 - 0 - 0  [nan] 0
34617 >scorpio-nn2(2): xboard
34617 >scorpio-nn2(2): protover 2
34617 >scorpio-nn1(3): xboard
34617 >scorpio-nn1(3): protover 2
34619 <scorpio-nn2(2): feature done=0
34619 >scorpio-nn2(2): accepted done
34620 <scorpio-nn1(3): feature done=0
34620 >scorpio-nn1(3): accepted done
34629 <scorpio-nn2(2): ht 4194304 X 16 = 64.0 MB
34629 <scorpio-nn2(2): eht 524288 X 8 = 8.0 MB
34629 <scorpio-nn2(2): pht 32768 X 24 = 0.8 MB
34629 <scorpio-nn2(2): treeht 335539200 X 40 = 12799.8 MB
34629 <scorpio-nn1(3): ht 4194304 X 16 = 64.0 MB
34629 <scorpio-nn1(3): eht 524288 X 8 = 8.0 MB
34629 <scorpio-nn1(3): pht 32768 X 24 = 0.8 MB
34629 <scorpio-nn1(3): treeht 335539200 X 40 = 12799.8 MB
34657 <scorpio-nn1(3): processors [16]
34661 <scorpio-nn2(2): processors [16]
34751 <scorpio-nn2(2): EgbbProbe 4.3 by Daniel Shawul
34752 <scorpio-nn1(3): EgbbProbe 4.3 by Daniel Shawul
180 egbbs loaded !      oading egbbs....
34981 <scorpio-nn1(3): Loading neural network : nets/net-0.pb
34985 <scorpio-nn1(3): Loading graph on /cpu:0
180 egbbs loaded !      oading egbbs....
34997 <scorpio-nn2(2): Loading neural network : nets/net-26.pb
35001 <scorpio-nn2(2): Loading graph on /cpu:0
35058 <scorpio-nn1(3): Neural network loaded !	
35058 <scorpio-nn1(3): loading_time = 0s
35089 <scorpio-nn2(2): Neural network loaded !	
35089 <scorpio-nn2(2): loading_time = 0s
Engine scorpio-nn2(2) did not start the chess protocol in time
Terminating process of engine scorpio-nn2(2)
Engine scorpio-nn1(3) did not start the chess protocol in time
Terminating process of engine scorpio-nn1(3)
Finished game 2 (scorpio-nn2 vs scorpio-nn1): * {No result}
Score of scorpio-nn1 vs scorpio-nn2: 0 - 0 - 0  [nan] 0
Xboard no problem game

Code: Select all

[Event "Computer Chess Game"]
[Site "dani-s-xp"]
[Date "2018.11.18"]
[Round "1"]
[White "Scorpio_2.8.9 MCTS+NN"]
[Black "Scorpio_2.8.9 MCTS+NN"]
[Result "*"]
[TimeControl "40/120"]
[Annotator "17. +0.55   14... -0.58"]

1. e4 e5 2. Nf3 Nf6 3. Nxe5 d6 4. Nf3 Nxe4 5. d4 d5 6. Bd3 Bd6 7. O-O O-O
8. c4 c6 9. cxd5 cxd5 10. Nc3 Nxc3 11. bxc3 Bg4 12. Rb1 b6 13. Rb5 Nd7 14.
h3 Be6 {-0.58/71 11} 15. Ng5 Nf6 {-0.42/71 5} 16. Qc2 Kh8 {-0.33/71 5} 17.
Bxh7 {+0.55/71 12} Bd7 {+0.24/71 5} 18. Rb1 {+0.00/72 0.1} g6 {+0.54/71 5}
19. Nxf7+ {+0.00/72 0.1} Rxf7 {+0.70/71 2.1} 20. Bxg6 {+0.00/72 0.1} Qg8
{+0.65/71 5} 21. Bxf7 {+0.52/71 6} Qxf7 {-0.45/71 2.2} 22. Qd2 {+0.78/71 6}
Qh7 {-0.42/71 5} 23. Qh6 {+0.34/71 6} Qxh6 {+0.51/71 5} 24. Bxh6
{-0.32/71 2.7} Rg8 {+0.76/71 5} 25. h4 {-0.36/71 6} Ne4 {+0.34/71 5}
*
According to the protocol, engines can take upto 1 hour to do initialization.

Daniel

User avatar
Guenther
Posts: 3067
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: what happened to scorpio NN in TCEC?

Post by Guenther » Sun Nov 18, 2018 11:06 am

Daniel Shawul wrote:
Sun Nov 18, 2018 10:49 am

According to the protocol, engines can take upto 1 hour to do initialization.

Daniel
I don't know why they don't use a compile which allows a longer start time (if this is the main problem)? It is an easy change.
I remember very well that LC0 also had that problem with too long start time in the first months.

OTH, 1 hour of course would be no fun and no one would ever use engines really taking up
one hour for initialising.
Current foe list count : [99]
http://rwbc-chess.de/chronology.htm

AndrewGrant
Posts: 490
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: what happened to scorpio NN in TCEC?

Post by AndrewGrant » Sun Nov 18, 2018 11:15 am

Daniel Shawul wrote:
Sun Nov 18, 2018 10:49 am
For a guy who complained and complained to get the previous admin out off TCEC to run away with his GUI, you should not be talking like that.
The previous admin did not run off, but instead was removed. Try to keep up.

Post Reply