My test for Orion 0.8 x64 pop-avx-fma Cerebrum

[Sylwy]

+378 Elo ! After a single match test, of course. Impressive !
Test conditions:
-TC=4'+2" (12-15 minutes per game)
-Hash=256 MB
-GUI: Arena 3.5.1
-Book: IM_4mvs.abk (8 plies book) for Orion 0.8 x64 pop-avx-fma Cerebrum and cheng2015.cb for Cheng 4.39_x64
-default settings for both engines
-1 thread-CPU=Intel i5-7400-3GHz (Kaby Lake)
-6-men Syzygy bases for Cheng 4.39 x64
-OS: Windows 10 Home.





Cheng 4.39 x64 has 2942 Elo points (CCRL Blitz).
Expected rating for Orion 0.8 x64 pop-avs-fma Cerebrum is around 3115 Elo points (CCRL Blitz).
+378 Elo face to Orion 0.7 x64 (handcrafted evaluation).
Orion 0.7 x64 (handcrafted evaluation) has 2737 Elo points (CCRL Blitz).

Let's wait for CCRL & CEGT results.

[Madeleine Birchfield]

Orion for TCEC Season 21?

[David Carteau]

Thanks Sylwy for the test !

It's a lot more than expected, and a lot more than in my own tests where I obtained more or less +200 elo (1000 games against 10 opponents at 40/2).

If this is confirmed with other tests, I would be very happy : that will mean that even being small (5M weights versus 10M for the "standard" NNUE architecture used in Stockfish), a neural network can learn very well !

[mar]

-6-men Syzygy bases for Cheng 4.39 x64

Cheng doesn't use Syzygy or any other TBs, but a very impressive result for Orion.

[Sylwy]

-6-men Syzygy bases for Cheng 4.39 x64

Cheng doesn't use Syzygy ....

This is true !

[Sylwy]

Thanks Sylwy for the test !

It's a lot more than expected, and a lot more than in my own tests where I obtained more or less +200 elo (1000 games against 10 opponents at 40/2).

My tests are for guidance only. Something systematic requires time and resources (like CCRL & CEGT). The results can sometimes be paradoxical. For example (in an own unpublished test) Chiron 4 x 64 crushed badly on Winter 0.9 x64. Although the (official) difference is only 3 Elo points.

[David Carteau]

My tests are for guidance only. Something systematic requires time and resources (like CCRL & CEGT). The results can sometimes be paradoxical. For example (in an own unpublished test) Chiron 4 x 64 crushed badly on Winter 0.9 x64. Although the (official) difference is only 3 Elo points.

Sorry Sylwy, didn't mean to say your tests are irrelevant (again, a big thank you!), but rather that my own tests might not be reliable at all since I used a set of opponents maybe too weak. My result could be very biased !