OKE - Opening Knowledge Engines

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Rebel
Posts: 4663
Joined: Thu Aug 18, 2011 10:04 am

OKE - Opening Knowledge Engines

Post by Rebel » Mon Jun 10, 2019 4:28 am

From another thread -
Rebel wrote:
Mon Jun 03, 2019 1:29 pm
I am disappointed, I was wondering if it made sense SF could build an opening book by itself using Multi-PV. First results (analysis time 100ms) looked promising 48.3% against the ProDeo book, 49.4% (analysis time 1000ms), now trying 2000ms but I think I am wasting my time looking at the moves it sometimes produces. Was hoping for a bug that could be fixed but looking at the responses that's unlikely.
And -
Rebel wrote:
Fri Jun 07, 2019 10:32 am
One idea would be to create a STS type of opening set rating engines with MEA from Ferdy.

Such as:
1.d4 and 1.e4 10 points
1.c4 and 1.Nf3 8 points
1.f4 6 points

etc.
And so I did, in search for the engine that can play the opening best. I created an opening set from 80,000+ games between (human) players of 2600+ elo for the first 4 moves resulting in 2836 epd positions and ran them with Ferdy's tool MEA creating a sorted list. So far I tested 26 engines and there are quite some surprises.

http://rebel13.nl/4-moves.html

#1. While Lc0 tops note that this is the slow CPU version and that 1 sec scores better than 5 secs :)

#2. That Rodent III is another surprise.

#3. That there isn't much difference between SF5 and SF10.

#4. Komodo 10 at 5 secs is almost equal to ProDeo 2.6

I am looking for candidates like Rodent.
Everybody is unique, except me.

User avatar
xr_a_y
Posts: 733
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: OKE - Opening Knowledge Engines

Post by xr_a_y » Mon Jun 10, 2019 7:57 am

Great stuff,

Can I ran the test on my engine by myself ?

Joerg Oster
Posts: 684
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: OKE - Opening Knowledge Engines

Post by Joerg Oster » Mon Jun 10, 2019 8:27 am

Interesting.
It looks like there is a LOT of room for improvement, in general.
Jörg Oster

User avatar
Rebel
Posts: 4663
Joined: Thu Aug 18, 2011 10:04 am

Re: OKE - Opening Knowledge Engines

Post by Rebel » Mon Jun 10, 2019 9:18 am

xr_a_y wrote:
Mon Jun 10, 2019 7:57 am
Great stuff,

Can I ran the test on my engine by myself ?
Yes, within a few days.
Everybody is unique, except me.

User avatar
xr_a_y
Posts: 733
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: OKE - Opening Knowledge Engines

Post by xr_a_y » Mon Jun 10, 2019 10:30 am

Rebel wrote:
Mon Jun 10, 2019 9:18 am
xr_a_y wrote:
Mon Jun 10, 2019 7:57 am
Great stuff,

Can I ran the test on my engine by myself ?
Yes, within a few days.
Great, can you give us the positions and the how-to please.

User avatar
Rebel
Posts: 4663
Joined: Thu Aug 18, 2011 10:04 am

Re: OKE - Opening Knowledge Engines

Post by Rebel » Mon Jun 10, 2019 4:21 pm

mea-epd.zip
(39.44 KiB) Downloaded 59 times
xr_a_y wrote:
Mon Jun 10, 2019 10:30 am
Rebel wrote:
Mon Jun 10, 2019 9:18 am
mea-epd.zip
(39.44 KiB) Downloaded 59 times
xr_a_y wrote:
Mon Jun 10, 2019 7:57 am
Great stuff,

Can I ran the test on my engine by myself ?
Yes, within a few days.
Great, can you give us the positions and the how-to please.
Positions for 2, 3 and 4 moves attached and then Download MEA. Else have a bit of patience.
Everybody is unique, except me.

User avatar
xr_a_y
Posts: 733
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: OKE - Opening Knowledge Engines

Post by xr_a_y » Mon Jun 10, 2019 4:59 pm

Rebel wrote:
Mon Jun 10, 2019 4:21 pm
mea-epd.zip
xr_a_y wrote:
Mon Jun 10, 2019 10:30 am
Rebel wrote:
Mon Jun 10, 2019 9:18 am
mea-epd.zip
xr_a_y wrote:
Mon Jun 10, 2019 7:57 am
Great stuff,

Can I ran the test on my engine by myself ?
Yes, within a few days.
Great, can you give us the positions and the how-to please.
Positions for 2, 3 and 4 moves attached and then Download MEA. Else have a bit of patience.
Thanks, I didn't wanna rush you, sorry.

User avatar
Rebel
Posts: 4663
Joined: Thu Aug 18, 2011 10:04 am

Re: OKE - Opening Knowledge Engines

Post by Rebel » Mon Jun 10, 2019 5:34 pm

http://rebel13.nl/2-moves.html

Same 30 engines, now using 2-moves.epd, only 404 positions.

#1. Look at Gideon from 1993, 5th place before SF10. A time before the internet, null-move was not invented, neither reductions, neither futility pruning etc. A time when a ply was a ply and you could count the number of moves of a combination and calculate the iteration an engine had to find the right move. A time also that all the pruning and reductions did not affect the evaluation function in a negative way.

#2. The once almighty Rybka at the bottom.

Next: 3-moves.epd (1221 positions) now running.
Everybody is unique, except me.

User avatar
Laskos
Posts: 9413
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: OKE - Opening Knowledge Engines

Post by Laskos » Mon Jun 10, 2019 5:49 pm

How did you pick the 2800 positions from 80,000 games? Were they in some way special? Did you use some analysis tool to pick them?
The results I see are strange. Lc0 positionally in the openings is far ahead of anything, so it managed to get to the top in your test too. But the rest seems a bit scrambled. What's that Rodent result? And many others. Also, what's that Lc0 worsens from 1s to 5s?
I estimate 1 standard deviation of the difference between engines to be about 800-1000 points in your scoring.

I have my custom made (hand-made) test-suite of 200 opening positions collected from a large database of human games (Elo above 2200, about 2 million games), but I had to pick by hand interesting positions according to the statistic of outcomes (and only marginally using engines' analysis to trim out tactical complications), it took some time to build it. Repeated 5 times to diminish the noise, I am using 1000 opening positions in this positional test suite with these much more normal results:

Code: Select all

4 i7 cores at 3.80GHz
RTX 2070 GPU

Openings1000.epd test-suite 
Sticks to the solution from 1s to 2s of thinking:

Lc0 v21.2 ID42524  757/1000
Lc0 v21.2 ID32930  727/1000
Lc0 v21.2 ID11261  723/1000
Stockfish_dev      574/1000
Houdini 6.03       558/1000
Komodo 13.02       556/1000
Xiphos 0.5         513/1000
Booot 6.3.1        494/1000
Andscacs 0.95      484/1000
Laser 1.7          480/1000
Ethereal 11.25     467/1000
Texel 1.07         419/1000
Rodent III         376/1000
Fruit 2.1          348/1000
BikJump 2.01       276/1000
Predateur 2.2.1    265/1000
I estimate my standard deviation to be about 10 points here. I frankly think that you have a lot of garbage in your suite.

User avatar
xr_a_y
Posts: 733
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: OKE - Opening Knowledge Engines

Post by xr_a_y » Mon Jun 10, 2019 6:15 pm

Rebel wrote:
Mon Jun 10, 2019 5:34 pm
http://rebel13.nl/2-moves.html

Same 30 engines, now using 2-moves.epd, only 404 positions.

#1. Look at Gideon from 1993, 5th place before SF10. A time before the internet, null-move was not invented, neither reductions, neither futility pruning etc. A time when a ply was a ply and you could count the number of moves of a combination and calculate the iteration an engine had to find the right move. A time also that all the pruning and reductions did not affect the evaluation function in a negative way.

#2. The once almighty Rybka at the bottom.

Next: 3-moves.epd (1221 positions) now running.

Code: Select all

Minic @100ms  per position is scoring 13240 for the 2-moves test.
Minic @1000ms per position is scoring 15076 for the 2-moves test.
This seems a little too strong for Minic no ?

Post Reply