What is wrong with Bright 0.4a2 under ChessGUI

[Matthias Gemuh]

Score is now 420:300 for BrightFRC.

At game/30sec, here are typical search depths:


[Event "Third World War"]
[Site "ChessGUI"]
[Date "2009.04.02"]
[Time "09:04:52"]
[Round "3"]
[White "BrightFRC-0.4a"]
[Black "Spike 1.2 Turin"]
[WhiteElo "1800"]
[BlackElo "2849"]
[Result "1-0"]
[TimeControl "30"]
[PlyCount "143"]
[Number "6"]
[WhiteType "program"]
[BlackType "program"]
[Variant "fischerandom"]
[SetUp "1"]
[FEN "bqnnrkrb/pppppppp/8/8/8/8/PPPPPPPP/BQNNRKRB w GEge -"]

{ Pentium 4 HT, 2.66 GHz, 512 MB RAM }
1.Nd3 { Eval = 20, Depth = 11, MoveTime = 00:00:00, } b5 { Eval = -28, Depth = 9, MoveTime = 00:00:00, }
2.g3 { Eval = 10, Depth = 11, MoveTime = 00:00:02, } Bxh1 { Eval = -13, Depth = 10, MoveTime = 00:00:00, }
3.Rxh1 { Eval = 6, Depth = 12, MoveTime = 00:00:01, } O-O { Eval = 17, Depth = 10, MoveTime = 00:00:00, }
4.Ne3 { ev=22,d=11,mt=00:00:00, } Nb6 { ev=12,d=10,mt=00:00:00, } 5.b3 { ev=20,d=10,mt=00:00:00, } Nc6 { ev=10,d=9,mt=00:00:00, }
6.f4 { ev=25,d=10,mt=00:00:00, } e6 { ev=33,d=8,mt=00:00:00, } 7.O-O-O { ev=22,d=9,mt=00:00:00, } d6 { ev=30,d=9,mt=00:00:00, }
8.Bc3 { ev=9,d=9,mt=00:00:00, } a5 { ev=44,d=10,mt=00:00:00, } 9.h4 { ev=-4,d=9,mt=00:00:00, } b4 { ev=30,d=10,mt=00:00:00, }
10.Bb2 { ev=14,d=11,mt=00:00:00, } a4 { ev=49,d=9,mt=00:00:00, } 11.h5 { ev=14,d=10,mt=00:00:00, } a3 { ev=44,d=9,mt=00:00:00, }
12.Ba1 { ev=12,d=10,mt=00:00:00, } f5 { ev=36,d=9,mt=00:00:00, } 13.h6 { ev=45,d=10,mt=00:00:00, } g6 { ev=18,d=8,mt=00:00:00, }
14.Bxh8 { ev=72,d=10,mt=00:00:00, } Kxh8 { ev=20,d=11,mt=00:00:00, }
15.Qa1 { ev=59,d=10,mt=00:00:00, } e5 { ev=33,d=11,mt=00:00:00, } 16.fxe5 { ev=65,d=10,mt=00:00:00, } dxe5 { ev=40,d=11,mt=00:00:00, }
17.Rh4 { ev=69,d=10,mt=00:00:00, } Qd8 { ev=0,d=10,mt=00:00:00, } 18.Nxb4 { ev=39,d=9,mt=00:00:00, } Nd4 { ev=19,d=11,mt=00:00:00, }
19.Rh2 { ev=42,d=10,mt=00:00:00, } Qd6 { ev=-23,d=9,mt=00:00:00, } 20.c3 { ev=57,d=10,mt=00:00:00, } c5 { ev=-53,d=9,mt=00:00:00, }
21.cxd4 { ev=66,d=10,mt=00:00:00, } cxb4 { ev=-60,d=9,mt=00:00:00, }
22.dxe5 { ev=49,d=9,mt=00:00:00, } Qxe5 { ev=-54,d=10,mt=00:00:00, }
23.Qxe5 { ev=49,d=10,mt=00:00:00, } Rxe5 { ev=-54,d=11,mt=00:00:00, }
24.Kb1 { ev=62,d=10,mt=00:00:00, } Rd8 { ev=-65,d=11,mt=00:00:00, } 25.Nc2 { ev=68,d=11,mt=00:00:00, } Nd5 { ev=-61,d=11,mt=00:00:00, }
26.Nd4 { ev=66,d=10,mt=00:00:00, } Rc8 { ev=-38,d=11,mt=00:00:00, } 27.Rc1 { ev=65,d=10,mt=00:00:00, } Rxc1 { ev=-51,d=11,mt=00:00:00, }
28.Kxc1 { ev=64,d=2,mt=00:00:00, } Nf6 { ev=-73,d=12,mt=00:00:00, } 29.e3 { ev=58,d=10,mt=00:00:00, } Rc5 { ev=-67,d=11,mt=00:00:00, }
30.Nc2 { ev=49,d=12,mt=00:00:00, } Rc6 { ev=-99,d=12,mt=00:00:00, } 31.Kb1 { ev=68,d=12,mt=00:00:00, } Rb6 { ev=-65,d=13,mt=00:00:00, }
32.Nd4 { ev=55,d=12,mt=00:00:00, } Kg8 { ev=-65,d=11,mt=00:00:00, } 33.Kc2 { ev=57,d=11,mt=00:00:00, } Rb8 { ev=-45,d=11,mt=00:00:00, }
34.Rf2 { ev=47,d=11,mt=00:00:00, } Rc8 { ev=-59,d=11,mt=00:00:00, } 35.Kd3 { ev=49,d=12,mt=00:00:00, } Rc1 { ev=-21,d=11,mt=00:00:00, }
36.Rf4 { ev=57,d=11,mt=00:00:00, } Nh5 { ev=-8,d=12,mt=00:00:00, } 37.Rh4 { ev=47,d=12,mt=00:00:00, } Nxg3 { ev=-38,d=12,mt=00:00:00, }
38.Ne2 { ev=79,d=12,mt=00:00:00, } Nxe2 { ev=-13,d=13,mt=00:00:00, }
39.Kxe2 { ev=60,d=14,mt=00:00:00, } Rc2 { ev=-43,d=14,mt=00:00:00, }
40.Rxb4 { ev=81,d=13,mt=00:00:00, } Rxa2 { ev=-12,d=14,mt=00:00:00, }
41.Rb8 { ev=54,d=12,mt=00:00:00, } Kf7 { ev=-38,d=5,mt=00:00:00, } 42.Rb7 { ev=75,d=13,mt=00:00:00, } Kg8 { ev=-6,d=13,mt=00:00:00, }
43.Rg7 { ev=83,d=12,mt=00:00:00, } Kh8 { ev=-6,d=13,mt=00:00:00, } 44.Ra7 { ev=78,d=13,mt=00:00:00, } Kg8 { ev=-53,d=14,mt=00:00:00, }
45.b4 { ev=78,d=12,mt=00:00:00, } Ra1 { ev=0,d=11,mt=00:00:00, } 46.b5 { ev=76,d=11,mt=00:00:00, } a2 { ev=-14,d=10,mt=00:00:00, }
47.Kd3 { ev=74,d=11,mt=00:00:00, } g5 { ev=-34,d=10,mt=00:00:00, } 48.Kd4 { ev=64,d=10,mt=00:00:00, } Rb1 { ev=-35,d=11,mt=00:00:00, }
49.Rxa2 { ev=64,d=11,mt=00:00:00, } Rxb5 { ev=-39,d=11,mt=00:00:00, }
50.Ra7 { ev=67,d=11,mt=00:00:00, } g4 { ev=-32,d=11,mt=00:00:00, } 51.Rg7 { ev=41,d=10,mt=00:00:00, } Kh8 { ev=-27,d=11,mt=00:00:00, }
52.Ra7 { ev=32,d=11,mt=00:00:00, } Kg8 { ev=0,d=12,mt=00:00:00, } 53.Rg7 { ev=42,d=12,mt=00:00:00, } Kh8 { ev=0,d=10,mt=00:00:00, }
54.Rg5 { ev=41,d=12,mt=00:00:00, } Rb6 { ev=-5,d=9,mt=00:00:00, } 55.Ke5 { ev=33,d=12,mt=00:00:00, } Rg6 { ev=-12,d=12,mt=00:00:00, }
56.Rxf5 { ev=26,d=12,mt=00:00:00, } Kg8 { ev=-31,d=11,mt=00:00:00, }
57.Ke4 { ev=104,d=12,mt=00:00:00, } g3 { ev=-118,d=10,mt=00:00:00, }
58.Rf1 { ev=107,d=12,mt=00:00:00, } Rxh6 { ev=-132,d=10,mt=00:00:00, }
59.Rg1 { ev=107,d=11,mt=00:00:00, } Rg6 { ev=-157,d=10,mt=00:00:00, }
60.Kf3 { ev=104,d=10,mt=00:00:00, } Kf7 { ev=-157,d=11,mt=00:00:00, }
61.Rxg3 { ev=112,d=10,mt=00:00:00, } Rf6 { ev=-161,d=9,mt=00:00:00, }
62.Kg4 { ev=114,d=10,mt=00:00:00, } Rg6 { ev=-154,d=9,mt=00:00:00, }
63.Kf4 { ev=121,d=11,mt=00:00:00, } Rf6 { ev=-166,d=10,mt=00:00:00, }
64.Ke5 { ev=121,d=11,mt=00:00:00, } Re6 { ev=-174,d=10,mt=00:00:00, }
65.Kd5 { ev=117,d=11,mt=00:00:00, } Rg6 { ev=-182,d=10,mt=00:00:00, }
66.Rxg6 { ev=195,d=12,mt=00:00:00, } Kxg6 { ev=-250,d=12,mt=00:00:00, }
67.d4 { ev=170,d=14,mt=00:00:00, } h5 { ev=-310,d=12,mt=00:00:00, } 68.Ke5 { ev=459,d=15,mt=00:00:00, } h4 { ev=-347,d=11,mt=00:00:00, }
69.Kf4 { ev=635,d=17,mt=00:00:00, } Kh5 { ev=-9999,d=13,mt=00:00:00, }
70.d5 { ev=644,d=16,mt=00:00:00, } h3 { ev=-9999,d=14,mt=00:00:00, }
71.Kg3 { ev=632,d=15,mt=00:00:00, } h2 { ev=-9999,d=10,mt=00:00:00, }
72.Kxh2 { ev=617,d=16,mt=00:00:00, }
1-0
{ Game Nr. 6 : BrightFRC-0.4a wins against Spike 1.2 Turin by GUI TB adjudication
GameDuration = 00:00:55, Finalposition 8/8/8/3P3k/8/4P3/7K/8 b - - }


[Event "Third World War"]
[Site "ChessGUI"]
[Date "2009.04.02"]
[Time "09:05:56"]
[Round "4"]
[White "Spike 1.2 Turin"]
[Black "BrightFRC-0.4a"]
[WhiteElo "2849"]
[BlackElo "1800"]
[Result "0-1"]
[TimeControl "30"]
[PlyCount "88"]
[Number "7"]
[BlackType "program"]
[WhiteType "program"]
[Variant "fischerandom"]
[SetUp "1"]
[FEN "qbbnnrkr/pppppppp/8/8/8/8/PPPPPPPP/QBBNNRKR w HFhf -"]

{ Pentium 4 HT, 2.66 GHz, 512 MB RAM }
1.c4 { Eval = 15, Depth = 10, MoveTime = 00:00:00, } c5 { Eval = -22, Depth = 11, MoveTime = 00:00:01, }
2.Ne3 { Eval = 26, Depth = 10, MoveTime = 00:00:00, } Nf6 { Eval = -8, Depth = 11, MoveTime = 00:00:01, }
3.b3 { Eval = 21, Depth = 10, MoveTime = 00:00:00, } d6 { Eval = -5, Depth = 10, MoveTime = 00:00:00, }
4.Bb2 { ev=45,d=9,mt=00:00:00, } Bd7 { ev=-1,d=10,mt=00:00:00, } 5.Bf5 { ev=74,d=9,mt=00:00:00, } Bxf5 { ev=-11,d=10,mt=00:00:00, }
6.Nxf5 { ev=67,d=10,mt=00:00:00, } Nc6 { ev=-23,d=11,mt=00:00:00, } 7.Bxf6 { ev=66,d=9,mt=00:00:00, } exf6 { ev=-35,d=11,mt=00:00:00, }
8.Nc2 { ev=56,d=10,mt=00:00:00, } Re8 { ev=-13,d=10,mt=00:00:00, } 9.Nce3 { ev=52,d=10,mt=00:00:00, } Bc7 { ev=-10,d=10,mt=00:00:01, }
10.Nd5 { ev=74,d=10,mt=00:00:00, } Qd8 { ev=-23,d=9,mt=00:00:00, } 11.e3 { ev=52,d=9,mt=00:00:00, } O-O { ev=-4,d=8,mt=00:00:00, }
12.d3 { ev=57,d=9,mt=00:00:00, } Re6 { ev=-14,d=9,mt=00:00:00, } 13.a3 { ev=78,d=8,mt=00:00:00, } Kh8 { ev=9,d=8,mt=00:00:00, }
14.f4 { ev=84,d=8,mt=00:00:00, } Nb8 { ev=-42,d=10,mt=00:00:02, } 15.Kf2 { ev=115,d=8,mt=00:00:00, } Nc6 { ev=-61,d=10,mt=00:00:00, }
16.b4 { ev=112,d=9,mt=00:00:00, } a6 { ev=-79,d=8,mt=00:00:00, } 17.bxc5 { ev=141,d=8,mt=00:00:00, } dxc5 { ev=-55,d=10,mt=00:00:00, }
18.Rb1 { ev=120,d=9,mt=00:00:00, } Qb8 { ev=-60,d=9,mt=00:00:00, } 19.Qb2 { ev=175,d=9,mt=00:00:00, } Na5 { ev=-77,d=9,mt=00:00:00, }
20.Rhf1 { ev=173,d=9,mt=00:00:00, } Kg8 { ev=-76,d=8,mt=00:00:00, } 21.Kg1 { ev=200,d=9,mt=00:00:00, } b6 { ev=-107,d=8,mt=00:00:00, }
22.Rf3 { ev=204,d=9,mt=00:00:00, } Qd8 { ev=-134,d=8,mt=00:00:00, } 23.Rg3 { ev=178,d=9,mt=00:00:00, } g6 { ev=-146,d=9,mt=00:00:00, }
24.Nh6 { ev=209,d=8,mt=00:00:00, } Kg7 { ev=-105,d=10,mt=00:00:00, }
25.Rh3 { ev=203,d=9,mt=00:00:00, } b5 { ev=-90,d=8,mt=00:00:00, } 26.Nxc7 { ev=174,d=9,mt=00:00:00, } Qxc7 { ev=-76,d=9,mt=00:00:00, }
27.cxb5 { ev=124,d=8,mt=00:00:00, } axb5 { ev=-63,d=9,mt=00:00:00, }
28.Qxb5 { ev=167,d=9,mt=00:00:00, } c4 { ev=-50,d=8,mt=00:00:00, } 29.d4 { ev=147,d=8,mt=00:00:00, } Nb3 { ev=-58,d=8,mt=00:00:00, }
30.Qb4 { ev=86,d=8,mt=00:00:00, } Ra8 { ev=-23,d=8,mt=00:00:00, } 31.d5 { ev=56,d=8,mt=00:00:00, } Rb6 { ev=45,d=9,mt=00:00:00, }
32.Qc3 { ev=2,d=9,mt=00:00:00, } Qc5 { ev=48,d=8,mt=00:00:00, } 33.d6 { ev=-6,d=8,mt=00:00:00, } Rxa3 { ev=90,d=8,mt=00:00:00, }
34.d7 { ev=-33,d=8,mt=00:00:00, } Rd6 { ev=106,d=8,mt=00:00:00, } 35.g4 { ev=-98,d=7,mt=00:00:00, } Rxd7 { ev=203,d=8,mt=00:00:00, }
36.g5 { ev=-173,d=8,mt=00:00:00, } Nd4 { ev=190,d=8,mt=00:00:00, } 37.Rb8 { ev=-194,d=8,mt=00:00:00, } fxg5 { ev=235,d=6,mt=00:00:00, }
38.Rg8 { ev=-261,d=8,mt=00:00:00, } Kf6 { ev=188,d=2,mt=00:00:00, } 39.Ng4 { ev=-261,d=7,mt=00:00:00, } Ke7 { ev=376,d=8,mt=00:00:00, }
40.Qe1 { ev=-442,d=7,mt=00:00:00, } Nb3 { ev=405,d=7,mt=00:00:00, } 41.Re8 { ev=-652,d=7,mt=00:00:00, } Kxe8 { ev=682,d=8,mt=00:00:00, }
42.Nf6 { ev=-781,d=8,mt=00:00:00, } Kd8 { ev=705,d=8,mt=00:00:00, } 43.Nxd7 { ev=-798,d=8,mt=00:00:00, } Kxd7 { ev=721,d=8,mt=00:00:00, }
44.Qd1 { ev=-814,d=7,mt=00:00:00, } Ke8 { ev=759,d=8,mt=00:00:00, }
0-1
{ Game Nr. 7 : BrightFRC-0.4a wins against Spike 1.2 Turin by GUI adjudication
GameDuration = 00:00:46, Finalposition 4k3/5p1p/6p1/2q3p1/2p2P2/rn2P2R/7P/3Q2K1 w - - }

[Matthias Gemuh]

I am trying to play through all 960 positions, alternating colours.
Time control is game/30sec and GUI is ChessGUI.
No loss on time for any of the 2 engines so far.

After 277 positions and 554 games, score is 326.5 : 227.5 for BrightFRC

Matthias.

Great, finally the bugfix seems to have a positive effect

After 1679 games, BrightFRC is leading by 1004.5 : 674.5 !!

Matthias.

[Matthias Gemuh]

I am trying to play through all 960 positions, alternating colours.
Time control is game/30sec and GUI is ChessGUI.
No loss on time for any of the 2 engines so far.

After 277 positions and 554 games, score is 326.5 : 227.5 for BrightFRC

Matthias.

Great, finally the bugfix seems to have a positive effect

After 1679 games, BrightFRC is leading by 1004.5 : 674.5 !!

Matthias.

Final resuult 1161.5 : 758.5 !!

No illegal moves.

Matthias.

[Allard Siemelink]

I am trying to play through all 960 positions, alternating colours.
Time control is game/30sec and GUI is ChessGUI.
Matthias.

Final resuult 1161.5 : 758.5 !!

No illegal moves.

Matthias.

Thanks for the test, I am glad the FRC bugs appear to be gone.
I gather bright 0.4 FRC is stronger then spike FRC after all.

[Spock]

Thanks for the test, I am glad the FRC bugs appear to be gone.
I gather bright 0.4 FRC is stronger then spike FRC after all.

Under my test conditions, very slightly, yes

http://www.computerchess.org.uk/ccrl/40 ... _pure.html

[Allard Siemelink]

Thanks for the test, I am glad the FRC bugs appear to be gone.
I gather bright 0.4 FRC is stronger then spike FRC after all.

Under my test conditions, very slightly, yes

http://www.computerchess.org.uk/ccrl/40 ... _pure.html

Thanks Ray, I am glad to see bright FRC having made its appearance on the CCRL FRC list.

I am not a statician, but if I apply HGM's formula to calculate the standard deviation for Matthias' result, I get:
sigma = 100%*.40 / sqrt(1920) ~= 1.0 %

Giving that bright scored 1161/1920~=60% against Spike, it means that Bright is better (head to head) with a confidence of 10 sigma or so (which is >99.99%)

In your tests, bright scored 50.5/100, if I apply the formula again, we get
sigma = 100%*.40 / sqrt(100) ~= 4%

I am not sure how the two different results correlate. With my limited statistics background, I would say that if we assume that
the bugfix did not alter bright's strength, the chances that bright scored 'only' 50.5% in you match against spike would be slim (around 1% or 2.5 sigma).

Perhaps it can be explained by the hypothesis that fixing the last FRC bug unexpectedly increased bright 0.4a3's strength significantly?

[Matthias Gemuh]

Thanks Ray, I am glad to see bright FRC having made its appearance on the CCRL FRC list.

I am not a statician, but if I apply HGM's formula to calculate the standard deviation for Matthias' result, I get:
sigma = 100%*.40 / sqrt(1920) ~= 1.0 %

Giving that bright scored 1161/1920~=60% against Spike, it means that Bright is better (head to head) with a confidence of 10 sigma or so (which is >99.99%)

In your tests, bright scored 50.5/100, if I apply the formula again, we get
sigma = 100%*.40 / sqrt(100) ~= 4%

I am not sure how the two different results correlate. With my limited statistics background, I would say that if we assume that
the bugfix did not alter bright's strength, the chances that bright scored 'only' 50.5% in you match against spike would be slim (around 1% or 2.5 sigma).

Perhaps it can be explained by the hypothesis that fixing the last FRC bug unexpectedly increased bright 0.4a3's strength significantly?

One important point is time control formats:
- "Game/x" versus "x/y repeating"

Another point is search time:
- "bullet" versus "blitz"

Matthias.

[Spock]

One important point is time control formats:
- "Game/x" versus "x/y repeating"

Another point is search time:
- "bullet" versus "blitz"

Matthias.

Yes different conditions.
As it happens, my result vs Spike is a "perfect " +0 ELO performance for Bright given Bright's overall rating. i.e., with Bright being +4 ELO stronger than Spike, a result of 50.5 - 49.5 is what you would expect.

Code: Select all

Bright 0.4a in CCRL 404FRC, 2009-04-03
Shredder GUI, ChessGUI by Matthias Gemuh, Ponder off, 5-men EGTB, 128MB hash, random openings with switched sides
Time control: Equivalent to 40 moves in 4 minutes on Athlon 64 X2 4600+ (2.4 GHz)
Tested by CCRL team, 2005-2009, http://computerchess.org.uk/ccrl/404FRC/

CCRL 404FRC main list:
Bright 0.4a is #9 with rating of 2822 ELO points (+16 -16),
based on 1700 games: 805 wins, 566 losses and 329 draws
Score: 57.0%, Average opponent: −57.1, Draws: 19.4%


Pairwise results:
     Opponent                    ELO     Score                  LOS   Perf
 - Naum 4 64-bit                 3079  13.5-86.5  (+8 -81=11)    0.0   -64
 - Deep Sjeng WC2008 64-bit 1CPU 2975  33.5-66.5  (+19-52=29)    0.0   +39
 - Shredder 11                   2974  29.0-71.0  (+19-61=20)    0.0    -6
 - Hiarcs 12.1                   2933  26.5-73.5  (+21-68=11)    0.0   -82
 - Glaurung 2.2 64-bit           2895  44.0-56.0  (+33-45=22)    0.0   +29
 - Fruit 051103                  2863  48.5-51.5  (+33-36=31)    0.0   +31
 - Loop 10.32f                   2856  36.5-63.5  (+23-50=27)    0.0   -59
 - Spike 1.2 Turin               2818  50.5-49.5  (+38-37=25)   67.7     0
 - Movei 00.8.438                2673  72.5-27.5  (+61-16=23)  100.0   +18
 - Frenzee Feb08 64-bit          2670  66.0-34.0  (+57-25=18)  100.0   -29
 - The Baron 2.23                2651  78.5-21.5  (+73-16=11)  100.0   +69
 - Pharaon 3.5.1                 2646  77.5-22.5  (+67-12=21)  100.0   +32
 - Jonny 2.83                    2617  78.5-21.5  (+71-14=15)  100.0   +24
 - Hamsters 0.7.1                2613  79.0-21.0  (+69-11=20)  100.0   +15
 - Hermann 2.4 64-bit            2593  77.0-23.0  (+72-18=10)  100.0    -2
 - Ufim 8.02                     2591  78.5-21.5  (+69-12=19)  100.0    -8
 - DanaSah 4.20                  2557  80.0-20.0  (+72-12=16)  100.0   -24

All but the last 200 games were played with the original Bright 0.4a

I did compare Bright's rating at FRC with our normal chess 40/4 list. I'm not sure if the scaling of the lists is such that you can compare the ELO directly, but when I look at Bright's rating relative to some common opponents around it - Glaurung, Fruit, Spike, Loop, Movei - it *appears* that Bright is about 30 ELO weaker at FRC compared to normal chess with neutral books.