My test for Orion 0.8 x64 pop-avx-fma Cerebrum

Discussion of computer chess matches and engine tournaments.

Moderator: Ras

User avatar
Sylwy
Posts: 5025
Joined: Fri Apr 21, 2006 4:19 pm
Location: IAȘI - the historical capital of MOLDOVA
Full name: Silvian Rucsandescu

My test for Orion 0.8 x64 pop-avx-fma Cerebrum

Post by Sylwy »

+378 Elo ! After a single match test, of course. Impressive !
Test conditions:
-TC=4'+2" (12-15 minutes per game)
-Hash=256 MB
-GUI: Arena 3.5.1
-Book: IM_4mvs.abk (8 plies book) for Orion 0.8 x64 pop-avx-fma Cerebrum and cheng2015.cb for Cheng 4.39_x64
-default settings for both engines
-1 thread-CPU=Intel i5-7400-3GHz (Kaby Lake)
-6-men Syzygy bases for Cheng 4.39 x64
-OS: Windows 10 Home.

Image

Image

Cheng 4.39 x64 has 2942 Elo points (CCRL Blitz).
Expected rating for Orion 0.8 x64 pop-avs-fma Cerebrum is around 3115 Elo points (CCRL Blitz).
+378 Elo face to Orion 0.7 x64 (handcrafted evaluation).
Orion 0.7 x64 (handcrafted evaluation) has 2737 Elo points (CCRL Blitz).

Let's wait for CCRL & CEGT results.
Madeleine Birchfield
Posts: 512
Joined: Tue Sep 29, 2020 4:29 pm
Location: Dublin, Ireland
Full name: Madeleine Birchfield

Re: My test for Orion 0.8 x64 pop-avx-fma Cerebrum

Post by Madeleine Birchfield »

Orion for TCEC Season 21?
David Carteau
Posts: 154
Joined: Sat May 24, 2014 9:09 am
Location: France
Full name: David Carteau

Re: My test for Orion 0.8 x64 pop-avx-fma Cerebrum

Post by David Carteau »

Thanks Sylwy for the test !

It's a lot more than expected, and a lot more than in my own tests where I obtained more or less +200 elo (1000 games against 10 opponents at 40/2).

If this is confirmed with other tests, I would be very happy : that will mean that even being small (5M weights versus 10M for the "standard" NNUE architecture used in Stockfish), a neural network can learn very well !
Download the Orion chess engine --- Train your NNUE with the Cerebrum library --- Contribute to the Nostradamus experiment !
mar
Posts: 2671
Joined: Fri Nov 26, 2010 2:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: My test for Orion 0.8 x64 pop-avx-fma Cerebrum

Post by mar »

Sylwy wrote: Thu Dec 03, 2020 4:48 pm -6-men Syzygy bases for Cheng 4.39 x64
Cheng doesn't use Syzygy or any other TBs, but a very impressive result for Orion.
User avatar
Sylwy
Posts: 5025
Joined: Fri Apr 21, 2006 4:19 pm
Location: IAȘI - the historical capital of MOLDOVA
Full name: Silvian Rucsandescu

Re: My test for Orion 0.8 x64 pop-avx-fma Cerebrum

Post by Sylwy »

mar wrote: Sat Dec 05, 2020 9:26 am
Sylwy wrote: Thu Dec 03, 2020 4:48 pm -6-men Syzygy bases for Cheng 4.39 x64
Cheng doesn't use Syzygy ....
This is true !

Image
User avatar
Sylwy
Posts: 5025
Joined: Fri Apr 21, 2006 4:19 pm
Location: IAȘI - the historical capital of MOLDOVA
Full name: Silvian Rucsandescu

Re: My test for Orion 0.8 x64 pop-avx-fma Cerebrum

Post by Sylwy »

David Carteau wrote: Sat Dec 05, 2020 8:59 am Thanks Sylwy for the test !

It's a lot more than expected, and a lot more than in my own tests where I obtained more or less +200 elo (1000 games against 10 opponents at 40/2).
My tests are for guidance only. Something systematic requires time and resources (like CCRL & CEGT). The results can sometimes be paradoxical. For example (in an own unpublished test) Chiron 4 x 64 crushed badly on Winter 0.9 x64. Although the (official) difference is only 3 Elo points.
David Carteau
Posts: 154
Joined: Sat May 24, 2014 9:09 am
Location: France
Full name: David Carteau

Re: My test for Orion 0.8 x64 pop-avx-fma Cerebrum

Post by David Carteau »

Sylwy wrote: Sat Dec 05, 2020 12:05 pm My tests are for guidance only. Something systematic requires time and resources (like CCRL & CEGT). The results can sometimes be paradoxical. For example (in an own unpublished test) Chiron 4 x 64 crushed badly on Winter 0.9 x64. Although the (official) difference is only 3 Elo points.
Sorry Sylwy, didn't mean to say your tests are irrelevant (again, a big thank you!), but rather that my own tests might not be reliable at all since I used a set of opponents maybe too weak. My result could be very biased !
Download the Orion chess engine --- Train your NNUE with the Cerebrum library --- Contribute to the Nostradamus experiment !