Official Release of Ethereal 12.50

[Laskos]

Note that I use a book called 8moves_v3 for Regression Testing. This book is typically very conservative. For the last few releases, (aside from some Contempt nonsense) I have under estimated the performance via this testing. This is likely due to the drawish nature of the opening book. I could be wrong, but +17 elo is a painful amount at this level, so even the lower end would make me happy.

Test for Standard Chess

Code: Select all

ELO   | 16.97 +- 3.41 (95%)
CONF  | 60.0+0.6s Threads=1 Hash=64MB
Games | N: 11269 W: 1873 L: 1323 D: 8073

Test for Fischer Random Chess

Code: Select all

ELO   | 40.77 +- 4.62 (95%)
CONF  | 60.0+0.6s Threads=1 Hash=64MB
Games | N: 8432 W: 2139 L: 1154 D: 5139

You specifically coded the eval for FRC or the openings are generally better?
I guess your 17 Elo points from 8moves_v3 would be some 25 Elo points on 2moves_v1 of the SF testing framework

Although the Fischer book does inflate elo more than 8moves_v3 does.

41 Elo points are probably some 15 Elo points above the expected, so it did improve more in FRC than in standard openings.

[Alayan]

Any engine that's not tuned/trained on FRC will miss out on a lot of FRC-patterns and lose strength. The Ethereal 12.50 improvements definitely are much better on FRC than standard chess because of this new tuning.

What's interesting is that adding FRC positions into the mix had no negative effect on standard chess performance. The FRC tunes passed non-regression in standard chess with positive score, and the mixed normal+FRC tuning set scored a general strength improvement on top of it.

This suggest that there is very little to lose in using FRC more widely for testing. A FRC engine that happens to play standard chess will get a better combined (FRC + standard) perf than a standard engine that happens to play FRC, so why not develop an engine with an eval that's good at generalizing across a wider range of positions.

[AndrewGrant]

Any engine that's not tuned/trained on FRC will miss out on a lot of FRC-patterns and lose strength. The Ethereal 12.50 improvements definitely are much better on FRC than standard chess because of this new tuning.

What's interesting is that adding FRC positions into the mix had no negative effect on standard chess performance. The FRC tunes passed non-regression in standard chess with positive score, and the mixed normal+FRC tuning set scored a general strength improvement on top of it.

This suggest that there is very little to lose in using FRC more widely for testing. A FRC engine that happens to play standard chess will get a better combined (FRC + standard) perf than a standard engine that happens to play FRC, so why not develop an engine with an eval that's good at generalizing across a wider range of positions.

I would even go so far as to say tuning with FRC has gained elo for Standard chess as well. In most cases of FRC based tunes, I would apply SPRT testing to the FRC and easily pass [0, 5], but then everytime I tried a [-3, 1] regression on Standard chess it passed, with positive values. Obviously I can't say that it gained elo, since [-3, 1] does not test that hypothesis, but it appears so.

+6.22 elo already since 12.50

[mwyoung]

i like Ethereal because chess isnt solved by Stockfish nnue !

Look at this:
[d]r5rk/1bq1bpp1/1p1ppn1p/pP5P/2P1P3/P1N2P2/1B1Q1P2/1K1R1BR1 w - - 0 19 bm Nd5

Stockfish nnue &Co didn´t have any clue.

Ethereal12.50-x64-popcnt:
37/65 03:28 1.652.261.891 7.911.000 +1,10 Dd2-e3 e6-e5 Sc3-d5 Lb7xd5 e4xd5 Sf6-d7 f3-f4 Le7-f6 Lf1-d3 Sd7-c5 f4xe5 Lf6xe5 Lb2xe5 d6xe5 Tg1-g4 Dc7-e7 Ld3-c2 Ta8-e8 f2-f3 a5-a4 Td1-g1 De7-d6 Tg1-g2 Dd6-f6 Tg2-g3 Te8-c8 De3-c3 Tc8-e8 Kb1-a2 Te8-a8 Tg3-g1 Ta8-e8 Dc3-e3 Te8-f8 Tg1-g2 Tf8-c8 De3-d2 Tc8-a8 Tg2-g1 Ta8-c8 Tg1-g3 Tc8-d8 Ka2-b1 Td8-e8 Tg3-g1 Te8-f8 Tg4-g3 e5-e4 Dd2-e3 Tf8-e8 f3xe4
38/60+ 05:39 2.525.504.260 7.445.000 +1,24 Sc3-d5
38/63+ 05:59 2.709.374.323 7.542.000 +1,38 Sc3-d5
38/60+ 07:05 3.135.051.741 7.363.000 +1,90 Sc3-d5
38/57 09:06 4.004.090.014 7.333.000 +1,91 Sc3-d5 Lb7xd5 e4xd5 e6-e5 f3-f4 e5-e4 Dd2-e3 Ta8-e8 Td1-c1 Sf6-d7 Lf1-h3 Le7-f6 Lh3xd7 Lf6xb2 Kb1xb2 Dc7xd7 De3xb6 Dd7-f5 Db6-e3 Te8-e7 Kb2-b3 f7-f6 Tg1-h1 Tg8-b8 a3-a4 Tb8-e8 Th1-h4 Df5-d7 c4-c5 d6xc5 Tc1xc5 Dd7-d6 Th4-h3 Kh8-g8 Th3-g3 Te7-d7 Kb3-a2 Te8-d8 De3-d4 Dd6xf4 b5-b6

"i like Ethereal because chess isnt solved by Stockfish nnue !"

New game Line
r5rk/1bq1bpp1/1p1ppn1p/pP5P/2P1P3/P1N2P2/1B1Q1P2/1K1R1BR1 w - - 0 1

Analysis by Stockfish 090820:

19.Nd5
White has a decisive advantage: +- (2.92 ++) Depth: 45/70 00:04:16 5986MN
(, 12.09.2020)

New game Line
r5rk/1bq1bpp1/1p1ppn1p/pP5P/2P1P3/P1N2P2/1B1Q1P2/1K1R1BR1 w - - 0 1

Analysis by Stockfish 090820:

19.Nd5 Bxd5 20.cxd5 Qd8 21.Rc1 exd5 22.exd5 Nh7 23.Qxh6 Bg5 24.Rxg5 Qxg5 25.Qxd6 Nf6 26.h6 Qxh6 27.Qxb6 Rgd8 28.Bc4 Rd7 29.Qd4 Qg5 30.Ka2 Re8 31.f4 Qg2 32.f3 Qxf3 33.Bb3 Re2 34.a4 Rxd5 35.Bxd5 Qxd5+ 36.Qxd5 Nxd5 37.Kb3 Re4 38.Rc4 Re3+ 39.Bc3 f6 40.Rc5 Nb6 41.Rc6 Nd5 42.Kc2 Nxc3 43.Rxc3 Rxc3+ 44.Kxc3 g5 45.fxg5 fxg5 46.b6
White is winning: +- (4.02 ++) Depth: 46/69 00:06:27 9101MN
(, 12.09.2020)