SPCC: Huge experimental test of Eman 6.60 learning-feature

[pohl4711]

Huge experimental test (3x 7000 games) of the Eman 6.60 learning-feature finished.

https://www.sp-cc.de/experiments.htm

(Perhaps you have to clear your browsercache or reload the website)

[Vinvin]

Very interesting, Thanks !

[Pi4Chess]

Huge experimental test (3x 7000 games) of the Eman 6.60 learning-feature finished.

https://www.sp-cc.de/experiments.htm

(Perhaps you have to clear your browsercache or reload the website)

Do your know if it's the same experience than with Brainlearn ?

[khalidomar]

You ran 21,000 games for 500 opening lines, which means each line was repeated 42 times. This is not enough for Eman to build useful experience.

Eman experience, like any other self-learning feature you can think of, needs a lot of data to produce tangible results.

I ran similar tests over a span of 1 year. Every time I receive the same results: Eman with experience is at least +50 ELO points stronger

In my tests, I use the same opening line(s) for at least 500-1000 games to collect experience data, then the actual test begins between two Eman's; one with experience and one without

After seeing your post I decided to repeat the test. Luckily I still have the last experience file generated for the following 3 lines (each line was repeated 1000 times, the TC used is 10s+1s)

Code: Select all

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 e6 6. g4 *

1. d4 Nf6 2. c4 e5 3. dxe5 Ng4 4. Bf4 Nc6 5. Nf3 Bb4+ 6. Nc3 *

1. c4 c5 2. Nc3 Nf6 3. Nf3 d5 4. cxd5 Nxd5 5. e4 Nb4 6. Bc4 Nd3+ 7. Ke2 Nf4+ 8. Kf1 Ne6 9. b4 cxb4 *

I started the test yesterday between Eman 6.60 and equivalent Stockfish (dev branch as of Nov 24, 2020). The TC used is 1m+1s

Right from the beginning, I could see that the results are going to be the same with Eman having a clear advantage. I stopped the test after 300 games with a score of +54 =239 -7 in Eman's favor (That is equivalent to +55 ELO points)

Capture.PNG

You can download the PGN from: EmanExp-Test-2020.12.06.pgn

If you would like to test the same experience file I used, then you can get it from: Eman.exp

This experience file is 17.5 MB only. Your experience file is large because it contains a lot of repeated positions. You should defragment it with Eman Tools and the size will be reduced dramatically

[khalidomar]

I forgot to mention that the test I performed yesterday was with 11 Threads for each engine, Ponder OFF, and 1024 MB Hash.

[Jouni]

Khalid one question: how can I disable NNUE in Eman!?

[khalidomar]

To disable NNUE in Eman you need to configure the NNUE Eval File parameter to the value <empty>

empty.png

[Jouni]

But isn't NNUE just learning which give up to +150 ELO to handtuning .