OKE - Opening Knowledge Engines

[Dann Corbit]

Rodent's score might well be an artifact caused by the fact that it likes fianchetto and finds/completes it more often than other engines.

Other opening-related bits and pieces are penalty for developing queen before minor pieces (but it is quite low and unlikely to influence the result), some code for pawn chains from King's Indian Defence/Attack (unlikely to kick in on move 4) and piece/square table asymmetry causing poor little mouse to dislike pawn on c2 (this one can have some influence).

The fianchetto part makes complete sense.
Even very strong engines struggle in the Indian openings (offense and defense)

[Rebel]

The engines that ere interesting here are those that out kicked their coverage. Rodent and Gideon stand out like sore thumbs. How do the differ in opening eval?

For Gideon one reason would be, as already stated earlier, no reductions, no futily pruning etc. --> no loss in the quality of eval.

That does not explain it for me.
Quite the opposite.

Speaking of quite, I have quite some examples for my statement. I would love to see a SF version that takes out most of the pruning and reductions except null-move and then run the tests again.

I think there is something that does not meet the eye.

Sure, it is certainly not the only reason but hard to define. Rodent and Gideon are both open source, the latter is described in detail.

[Ovyron]

What I mean is there are 700 million games here:
https://database.lichess.org/

I have a parser that will collect win loss draw (and any other available information) from the games.
You can (of course) filter for Elo, but I do not think that is as important as most people think because the bad moves will still lose and the good moves will still win.

Right, what you want to measure is the move's elo performance, which GUIs are able to do with ease. So, you'd want to do something like this:

1. Download all those games. Or all the games from all sources that you want.}

2. Create an opening book from all of them in software like Chessbase or InfinityChess.

3. Take a look at the "Perf." column on the GUI, Rebel would like to give the biggest bonus to the moves with the best elo performance.

And that's about it, I doubt 2.Nf3 or 2.g3 would have a radically different elo performance, so they would automatically award about the same points, without needing manual tweaking.

[Rebel]

OKE released - http://rebel13.nl/rebel13/oke.html

Whatever the verdict at least the larger EPD sets are useful for eng-eng testing.

[Ferdy]

OKE released - http://rebel13.nl/rebel13/oke.html

Whatever the verdict at least the larger EPD sets are useful for eng-eng testing.

Thanks this is interesting for engines.

I will be releasing new mea next week with option to set engine options for uci engines, although it can be set when using polyglot. There is also a bug that I need to address when the move is a promotion, that is a8=Q=100, the 2 equal signs would confuse mea. Your opening sets are not affected by this as there are no promotion moves on the top moves.

[Rebel]

OKE released - http://rebel13.nl/rebel13/oke.html

Whatever the verdict at least the larger EPD sets are useful for eng-eng testing.

Thanks this is interesting for engines.

I will be releasing new mea next week with option to set engine options for uci engines, although it can be set when using polyglot. There is also a bug that I need to address when the move is a promotion, that is a8=Q=100, the 2 equal signs would confuse mea. Your opening sets are not affected by this as there are no promotion moves on the top moves.

Ah, excellent news. Perhaps you can printf the end of the log-file on the screen before exit.

Code: Select all

[ 2019-06-14 10:23:32,463 -     INFO ] Time allocation  : GOOD!!
[ 2019-06-14 10:23:32,463 -     INFO ] ExpectedTime     : 404.0s
[ 2019-06-14 10:23:32,463 -     INFO ] ActualTime       : 369.1s
[ 2019-06-14 10:23:32,463 -     INFO ] TimeMargin/pos   : 0.2s
[ 2019-06-14 10:23:32,463 -     INFO ] TimeMarginTotal  : 80.8s
[ 2019-06-14 10:23:32,463 -     INFO ] Writing analysis results ...
[ 2019-06-14 10:23:32,759 -     INFO ] Done!!
[ 2019-06-14 10:23:32,759 -     INFO ] move ProDeo_2.6_WB_log.txt to log dir

and

Code: Select all

[ 2019-06-09 15:40:35,494 -     INFO ] Time allocation  : BAD!! spending less time
[ 2019-06-09 15:40:35,494 -     INFO ] ExpectedTime     : 17016.0s
[ 2019-06-09 15:40:35,494 -     INFO ] ActualTime       : 12864.0s
[ 2019-06-09 15:40:35,494 -     INFO ] TimeMargin/pos   : 0.2s
[ 2019-06-09 15:40:35,494 -     INFO ] TimeMarginTotal  : 567.2s
[ 2019-06-09 15:40:35,509 -     INFO ] Writing analysis results ...
[ 2019-06-09 15:40:35,775 -     INFO ] Done!!
[ 2019-06-09 15:40:35,775 -     INFO ] move Gideon_log.txt to log dir

Things like these easily escape attention.

Code: Select all

[ 2019-06-08 11:17:53,911 -     INFO ] Time allocation  : GOOD!!
[ 2019-06-08 11:17:53,911 -     INFO ] ExpectedTime     : 410.4s
[ 2019-06-08 11:17:53,911 -     INFO ] ActualTime       : 664.6s
[ 2019-06-08 11:17:53,911 -     INFO ] TimeMargin/pos   : 0.2s
[ 2019-06-08 11:17:53,911 -     INFO ] TimeMarginTotal  : 820.8s
[ 2019-06-08 11:17:53,911 -     INFO ] Writing analysis results ...
[ 2019-06-08 11:17:54,191 -     INFO ] Done!!
[ 2019-06-08 11:17:54,191 -     INFO ] move Andscacs_0.93_log.txt to log dir

I would not rate this as good

[Rebel]

I will be releasing new mea next week with option to set engine options for uci engines, although it can be set when using polyglot. There is also a bug that I need to address when the move is a promotion, that is a8=Q=100, the 2 equal signs would confuse mea. Your opening sets are not affected by this as there are no promotion moves on the top moves.

If it's not too much trouble I have 2 other wishes as well.

#1. The log file is great to extract an EPD from with bm, ce and acd. Is it possible for you to that automatically in the new version?

#2. Is it possible to add Muli-PV to MEA? Much more work involved I suppose.

[Ferdy]

I will be releasing new mea next week with option to set engine options for uci engines, although it can be set when using polyglot. There is also a bug that I need to address when the move is a promotion, that is a8=Q=100, the 2 equal signs would confuse mea. Your opening sets are not affected by this as there are no promotion moves on the top moves.

If it's not too much trouble I have 2 other wishes as well.

#1. The log file is great to extract an EPD from with bm, ce and acd. Is it possible for you to that automatically in the new version?

#2. Is it possible to add Muli-PV to MEA? Much more work involved I suppose.

Using the new option

Code: Select all

--eoption "multipv=3, threads=1, hash=256"

The ce and acd are from last info from multipv 1.
The bm is always from the bestmove info sent by engine and not from the first move of pv in multipv 1 line.

Code: Select all

...
[ 2019-06-15 15:17:19,914 -     INFO ] eoption: ['multipv=3', ' threads=1', ' hash=256']
[ 2019-06-15 15:17:19,914 -     INFO ] setoption name multipv value 3
[ 2019-06-15 15:17:19,914 -     INFO ] setoption name threads value 1
[ 2019-06-15 15:17:19,914 -     INFO ] setoption name hash value 256
[ 2019-06-15 15:17:19,914 -    DEBUG ] >> isready
[ 2019-06-15 15:17:20,039 -    DEBUG ] << readyok
[ 2019-06-15 15:17:20,039 -     INFO ] 

[ 2019-06-15 15:17:20,039 -     INFO ] Pos 1
[ 2019-06-15 15:17:20,039 -     INFO ] id tony.pos.01
[ 2019-06-15 15:17:20,039 -     INFO ] FEN: 1r2r1k1/1pqbbppp/p2p1n2/4p3/P3PP2/2N1BB2/1PP2QPP/R4R1K b - - 0 1
[ 2019-06-15 15:17:20,039 -     INFO ] Solutions: Rbc8=10, h6=9, exf4=8, Qc4=7, Kh8=6, Bc6=5, Ra8=4, Qc6=4, Qc8=2, Be6=1
[ 2019-06-15 15:17:20,039 -    DEBUG ] >> ucinewgame
[ 2019-06-15 15:17:20,039 -    DEBUG ] >> position fen 1r2r1k1/1pqbbppp/p2p1n2/4p3/P3PP2/2N1BB2/1PP2QPP/R4R1K b - - 0 1
[ 2019-06-15 15:17:20,039 -    DEBUG ] >> go movetime 5000
[ 2019-06-15 15:17:20,071 -    DEBUG ] << info depth 1 seldepth 1 multipv 1 score cp 41 nodes 141 nps 141000 tbhits 0 time 1 pv f6g4

...

[ 2019-06-15 15:17:24,595 -    DEBUG ] << info depth 19 seldepth 24 multipv 3 score cp -9 nodes 6674068 nps 1472329 hashfull 166 tbhits 0 time 4533 pv b7b5 a4b5 d7b5 c3b5 a6b5 f4e5 d6e5 g2g4 h7h6 g4g5 h6g5 e3g5 b8c8 c2c3 c8a8 g5e3 a8a1 f1a1 b5b4 c3b4 e7b4 f2g2
[ 2019-06-15 15:17:24,845 -    DEBUG ] << info depth 20 seldepth 32 multipv 1 score cp 26 nodes 7029717 nps 1473426 hashfull 180 tbhits 0 time 4771 pv b8c8 a1d1 g8h8 f4f5 b7b5 e3g5 d7c6 a4b5 a6b5 g5f6 e7f6 f3e2 c7b8 b2b4 b8b7 f2f3 e8d8 d1a1 d6d5 e4d5 c6d5 c3d5 b7d5 f3d5 d8d5
[ 2019-06-15 15:17:24,845 -    DEBUG ] << info depth 20 seldepth 30 multipv 2 score cp 3 nodes 7029717 nps 1473426 hashfull 180 tbhits 0 time 4771 pv c7c4 f4e5 d6e5 c3d5 f6d5 e4d5 f7f5 d5d6 e7f6 e3a7 e5e4 f3e2 c4b4 a7b8 e8b8 c2c3 b4d6 e2c4 g8h8 f1d1 d6e7 f2e3 b8c8 c4b3 d7e6 b3e6 e7e6 e3a7 e4e3 a7b7
[ 2019-06-15 15:17:24,845 -    DEBUG ] << info depth 19 seldepth 24 multipv 3 score cp -9 nodes 7029717 nps 1473426 hashfull 180 tbhits 0 time 4771 pv b7b5 a4b5 d7b5 c3b5 a6b5 f4e5 d6e5 g2g4 h7h6 g4g5 h6g5 e3g5 b8c8 c2c3 c8a8 g5e3 a8a1 f1a1 b5b4 c3b4 e7b4 f2g2
[ 2019-06-15 15:17:25,062 -    DEBUG ] << info depth 20 seldepth 32 multipv 1 score cp 26 nodes 7371493 nps 1474003 hashfull 190 tbhits 0 time 5001 pv b8c8 a1d1 g8h8 f4f5 b7b5 e3g5 d7c6 a4b5 a6b5 g5f6 e7f6 f3e2 c7b8 b2b4 b8b7 f2f3 e8d8 d1a1 d6d5 e4d5 c6d5 c3d5 b7d5 f3d5 d8d5
[ 2019-06-15 15:17:25,062 -    DEBUG ] << info depth 20 seldepth 30 multipv 2 score cp 3 nodes 7371493 nps 1474003 hashfull 190 tbhits 0 time 5001 pv c7c4 f4e5 d6e5 c3d5 f6d5 e4d5 f7f5 d5d6 e7f6 e3a7 e5e4 f3e2 c4b4 a7b8 e8b8 c2c3 b4d6 e2c4 g8h8 f1d1 d6e7 f2e3 b8c8 c4b3 d7e6 b3e6 e7e6 e3a7 e4e3 a7b7
[ 2019-06-15 15:17:25,062 -    DEBUG ] << bestmove b8c8 ponder a1d1
[ 2019-06-15 15:17:25,062 -     INFO ] elapsed(ms) since go: 5022
[ 2019-06-15 15:17:25,062 -     INFO ] bestmove: Rbc8
[ 2019-06-15 15:17:25,062 -     INFO ] Top 1 move!!
[ 2019-06-15 15:17:25,062 -     INFO ] Score for this test: 10
[ 2019-06-15 15:17:25,062 -     INFO ] Total Score update: 10 / 10 (1.000)
[ 2019-06-15 15:17:25,062 -     INFO ] 1r2r1k1/1pqbbppp/p2p1n2/4p3/P3PP2/2N1BB2/1PP2QPP/R4R1K b - - bm Rbc8; ce 26; acd 20;
[ 2019-06-15 15:17:25,062 -     INFO ] 

Is the line with epd, bm, ce, acd enough? What output format do you like when multipv is more than 1.

[Rebel]

1r2r1k1/1pqbbppp/p2p1n2/4p3/P3PP2/2N1BB2/1PP2QPP/R4R1K b - - bm Rbc8; ce 26; acd 20;

Is the line with epd, bm, ce, acd enough?

Excellent!

Code: Select all

mea.exe --engine "sf10.exe" --protocol uci --name "Stockfish10" --hash %HASH% --threads 4 --rating 3495 --epd %EPD% --movetime %MT% --log 1 --output %OUTPUT%

And the filename as given by the user with --name ,thus Stockfish10.epd

What output format do you like when multipv is more than 1.

If possible something like this:

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - id MultiPV=1; bm e4; ce=30; acd=20;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - id MultiPV=2; bm d4; ce=20; acd=20;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - id MultiPV=3; bm Nf3; ce=10; acd=20;

So 3 epd records.

[Ferdy]

1r2r1k1/1pqbbppp/p2p1n2/4p3/P3PP2/2N1BB2/1PP2QPP/R4R1K b - - bm Rbc8; ce 26; acd 20;

Is the line with epd, bm, ce, acd enough?

Excellent!

Code: Select all

mea.exe --engine "sf10.exe" --protocol uci --name "Stockfish10" --hash %HASH% --threads 4 --rating 3495 --epd %EPD% --movetime %MT% --log 1 --output %OUTPUT%

And the filename as given by the user with --name ,thus Stockfish10.epd

All right will do that.
But perhaps [epd name]_[engine name].epd. Can be 2-moves_Stockfish10.epd if the epd is 2-moves.epd

What output format do you like when multipv is more than 1.

If possible something like this:

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - id MultiPV=1; bm e4; ce=30; acd=20;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - id MultiPV=2; bm d4; ce=20; acd=20;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - id MultiPV=3; bm Nf3; ce=10; acd=20;

So 3 epd records.

This one

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - id MultiPV=1; bm e4; ce=30; acd=20;

is not a valid epd. We can modify it to ce 30; acd 20;
But if you really like that format with ce=30, I will do that.

If epd is 2-moves.epd, that epd results can be saved in 2-moves_multipv[multipv number]_Stockfish10.epd.
Then we can also make the id's unique, can be

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - id "2-moves.epd pos 1 MultiPV=1"; bm e4; ce=30; acd=20;
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - id "2-moves.epd pos 1 MultiPV=2"; bm d4; ce=20; acd=20;
...
[piece locations] [side] [castle rights] [ep] id "2-moves.epd pos 2 MultiPV=1"; ...
...