Komodo - Stockfish (1.000 games) ... FEOBOS v5.0 Test-Set

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Rebel, chrisw

Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Komodo - Stockfish (1.000 games) ... FEOBOS v5.0 Test-Set

Post by Frank Quisinsky »

Hi there,

I have to test the new FEOBOS v5.0 Test-Set and using the current versions of Stockfish and Komodo for it.

Hint:
I am using one Intel i7 4.0 Ghz system only. For 1.000 games around 8-9 days I think.

40 moves in 10 minutes, ponder = off, 512Mb for hash-tables, 4-pieces ... just the older FCP Rating List conditions.

If you like to follow:
http://www.amateurschach.de/ftptrigger/ ... t-set.html

Best lines to each of the 500 ECO codes, included in FEOBOS v5.0 opening book selected as Test-Set.

= 500 with black and white pieces = 1.000 games.
2x A00, 2x A01, 2x A02 ... I am not the test-set expert but I am sure this match will produced a very exactly final result.

The new FEOBOS v5.0 Test-Set can be found on my webpage under:

Frank's Chess Page
http://www.amateurschach.de

Have fun with it. It's just a test for the work Klaus Wlotzka and I do with FEOBOS databases, not more, not less.

Ah, ASMFish is playing without contempt. Komodo with contempt = 10, standard setting. For ASMFish contempt isn't necessary with the database I am using. Have a look in the message I wrote to FEOBOS v5.0 Test-Set in TalkChess main forum.

Best
Frank
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: After 200 games, A00-A99 ... stats!

Post by Frank Quisinsky »

Hi there,

A00-A99 is ready = 200 games!

Games can be download under:
http://www.amateurschach.de/download/wa ... komodo.zip (0.33Mb).

Included in download file the current FEOBOS v5.02 Test-Set (*.pgn, *.epd, readme.txt). Changes I do in "Test-Set" can be found in readme.

Again this is a test for FEOBOS only.
Important here to have clear positions from A00-E99. Often a code goes after end position with move transposition in another. So I have to search the second best variant in FEOBOS database.

Note:
In all of the 500 positions are Queens on board. I changed that too ... again information can be found in the readme.

Code: Select all

Games        :    200 (finished)

White Wins   :     46 (23.0 %)
Black Wins   :     27 (13.5 %)
Draws        :    127 (63.5 %)
Unfinished   :      0

White Perf.  : 54.8 %
Black Perf.  : 45.2 %

ECO A =    200 Games (100.0 %)
ECO B =      0 Games ( 0.0 %)
ECO C =      0 Games ( 0.0 %)
ECO D =      0 Games ( 0.0 %)
ECO E =      0 Games ( 0.0 %)

Code: Select all

Individual statistics:

1 ASMFishW 20170522 BMI2 x64: 3240  200 (+ 59,=127,- 14), 61.2 %

Komodo 11.01 x64              : 200 (+ 59,=127,- 14), 61.3 %

2 Komodo 11.01 x64          : 3160  200 (+ 14,=127,- 59), 38.8 %

ASMFishW 20170522 BMI2 x64    : 200 (+ 14,=127,- 59), 38.8 %

Code: Select all

Games   = 200   ( no result = 0,  FEN tags = 0 )
Players = 2   ( clusters = 1 )
Date Range: 2017.06.14 - 2017.06.16

Games with:  WhiteElo = 0   BlackElo = 0   BothElos = 0

White Wins = 46 ( 23.0 % )
Draws      = 127 ( 63.5 % )
Black Wins = 27 ( 13.5 % )
White Pct = 54.75 %
Black Pct = 45.25 %

ECO:  Total = 200  A: 200  B: 0  C: 0  D: 0  E: 0
PlyCount:  Total = 200  Range: 50-590  Average = 171.0  StdDev = 73.84

finished: be sure to rename/copy outSummary
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Shortest, Komodo lost in 44, B08 on 1.225 (ranking)!

Post by Frank Quisinsky »

With FEOBOS positions we can be sure that the opening is OK. Engine testing on a very high level ... engines tell us which positions we have to give (FEOBOS).

OK we have a B08 short game, mate after 44 moves!
Now we can looking in FEOBOS excel file.

B08 Pos 06174
Houdini: 0.10 = Nd2, Ply 22
Shredder: 0.17 = h4, Ply 27
Fire: 0.26 = Nd2, Ply 22
Wasp: 0.37 = 0-0, Ply 23
Chiron: 0.46 = Qd3, Ply 24

At the moment 5 engines added in FEOBOS.
Klaus is working on rankings for each of the positions. Will be in one of the next Excel files available.

Pos. 06174 B08 is on 1.225 / 26.146

A very high ranking, should be clear because ...
5 engines analyzed ... and 4x another first "best" move.
Optimal for a test-set position and the ranking is high.

Note:
After the first Booot analyzes (engine 6) we have a A04 Position with 6 different best lines ... the first place in ranking.

:-)

Note: The current test-set is good but not perfect because 5 engines added only in FEOBOS. 8 others will be soon available. With more engines the test-set will be better because the ranking is more exactly.

Ah, here the game Komodo have no chance! Very perfect attacking game by Stockfish.

[pgn][Event "FEOBOS, 40/10, p=off, ht512, i7 4GHz"]
[Site "Trier"]
[Date "2017.06.16"]
[Round "217.1"]
[White "ASMFishW 20170522 BMI2 x64"]
[Black "Komodo 11.01 x64"]
[Result "1-0"]
[ECO "B08"]
[PlyCount "87"]
[EventDate "2017.??.??"]

1. d4 Nf6 2. Nf3 g6 3. Nc3 Bg7 4. e4 d6 5. Be3 a6 6. a4 O-O 7. Be2 b6 8. Nd2 {
[%eval 14,26] [%emt 0:00:52]} Bb7 {[%eval 3,22] [%emt 0:00:13]} 9. h4 {[%eval
4,26] [%emt 0:00:19] (O-O)} h5 {[%eval -14,23] [%emt 0:00:34] (c5)} 10. g4 {
[%eval 96,24] [%emt 0:00:07] (O-O)} hxg4 {[%eval 63,22] [%emt 0:00:14]} 11.
Bxg4 {[%eval 129,25] [%emt 0:00:16]} c5 {[%eval 65,24] [%emt 0:00:28]} 12. d5 {
[%eval 173,23] [%emt 0:00:06]} Nbd7 {[%eval 94,24] [%emt 0:00:40]} 13. h5 {
[%eval 174,26] [%emt 0:00:15]} Ne5 {[%eval 133,24] [%emt 0:00:38]} 14. Be2 {
[%eval 210,24] [%emt 0:00:05]} Re8 {[%eval 122,23] [%emt 0:00:15] (gxh5)} 15.
f4 {[%eval 262,23] [%emt 0:00:08]} Ned7 {[%eval 123,22] [%emt 0:00:05]} 16.
hxg6 {[%eval 299,24] [%emt 0:00:08] (h6)} fxg6 {[%eval 109,21] [%emt 0:00:05]}
17. Bg4 {[%eval 331,23] [%emt 0:00:05]} Nxg4 {[%eval 177,24] [%emt 0:00:44]
(e6)} 18. Qxg4 {[%eval 354,24] [%emt 0:00:07]} e6 {[%eval 272,23] [%emt 0:00:
11]} 19. Qxg6 {[%eval 408,26] [%emt 0:00:16] (dxe6)} exd5 {[%eval 315,21]
[%emt 0:00:09]} 20. f5 {[%eval 437,25] [%emt 0:00:03]} Nf8 {[%eval 406,22]
[%emt 0:00:18] (d4)} 21. Qg4 {[%eval 604,23] [%emt 0:00:11]} d4 {[%eval 442,24]
[%emt 0:00:21]} 22. Bh6 {[%eval 626,25] [%emt 0:00:15] (Tg1)} Qf6 {[%eval 472,
22] [%emt 0:00:18]} 23. Ne2 {[%eval 636,25] [%emt 0:00:06]} Re7 {[%eval 556,24]
[%emt 0:00:44] (Tad8)} 24. Bg5 {[%eval 752,23] [%emt 0:00:09]} Qe5 {[%eval 583,
22] [%emt 0:00:08] (Df7)} 25. Bf4 {[%eval 854,24] [%emt 0:00:10] (Lxe7)} Qf6 {
[%eval 714,23] [%emt 0:00:09]} 26. Rh6 {[%eval 869,22] [%emt 0:00:04]} Nh7 {
[%eval 816,26] [%emt 0:00:17]} 27. Rxf6 {[%eval 906,23] [%emt 0:00:10]} Nxf6 {
[%eval 894,26] [%emt 0:00:13]} 28. Qg6 {[%eval 919,24] [%emt 0:00:09]} Bxe4 {
[%eval 902,23] [%emt 0:00:07]} 29. Bh6 {[%eval 954,25] [%emt 0:00:12]} Bxc2 {
[%eval 960,24] [%emt 0:00:19] (Lxf5)} 30. Bxg7 {[%eval 1013,25] [%emt 0:00:28]}
Rxg7 {[%eval 1038,25] [%emt 0:00:18]} 31. Qxf6 {[%eval 1025,23] [%emt 0:00:05]}
Re8 {[%eval 1063,23] [%emt 0:00:12] (Tf8)} 32. Kf2 {[%eval 1215,26] [%emt 0:00:
38]} Bxf5 {[%eval 1090,23] [%emt 0:00:11]} 33. Nf4 {[%eval 1326,24] [%emt 0:00:
40]} Bh7 {[%eval 1188,23] [%emt 0:00:23] (Lg6)} 34. Ne6 {[%eval 12797,27]
[%emt 0:00:12]} Rg6 {[%eval 1250,26] [%emt 0:00:30]} 35. Rg1 {[%eval 12831,28]
[%emt 0:00:05]} Rxg1 {[%eval 1717,22] [%emt 0:00:06]} 36. Kxg1 {[%eval 32726,
34] [%emt 0:00:29]} Rxe6 {[%eval 1800,22] [%emt 0:00:03]} 37. Qxe6+ {[%eval
32746,34] [%emt 0:00:35]} Kf8 {[%eval 1843,20] [%emt 0:00:12] (Kg7)} 38. Qf6+ {
[%eval 32748,47] [%emt 0:00:27] (Dh6+)} Kg8 {[%eval 25000,22] [%emt 0:00:05]
(Ke8)} 39. Nc4 {[%eval 32756,72] [%emt 0:00:28] (Sf3)} d3 {[%eval 32758,29]
[%emt 0:00:00]} 40. Nxd6 {[%eval 32758,78] [%emt 0:00:12]} d2 {[%eval 32760,99]
[%emt 0:00:00]} 41. Qf7+ {[%eval 32760,94] [%emt 0:00:18]} Kh8 {[%eval 32762,5]
[%emt 0:00:00]} 42. Qf8+ {[%eval 32762,97] [%emt 0:00:04]} Bg8 {[%eval 32764,5]
[%emt 0:00:00]} 43. Nf7+ {[%eval 32764,127] [%emt 0:00:28]} Kh7 {[%eval 32766,
5] [%emt 0:00:00]} 44. Qh6# {[%eval 32766,127] [%emt 0:00:00]} 1-0
[/pgn]

Best
Frank
beram
Posts: 1187
Joined: Wed Jan 06, 2010 3:11 pm

Re: Shortest, Komodo lost in 44, B08 on 1.225 (ranking)!

Post by beram »

Frank Quisinsky wrote:With FEOBOS positions we can be sure that the opening is OK. Engine testing on a very high level ... engines tell us which positions we have to give (FEOBOS).

OK we have a B08 short game, mate after 44 moves!
Now we can looking in FEOBOS excel file.

B08 Pos 06174
Houdini: 0.10 = Nd2, Ply 22
Shredder: 0.17 = h4, Ply 27
Fire: 0.26 = Nd2, Ply 22
Wasp: 0.37 = 0-0, Ply 23
Chiron: 0.46 = Qd3, Ply 24

At the moment 5 engines added in FEOBOS.
Klaus is working on rankings for each of the positions. Will be in one of the next Excel files available.

Pos. 06174 B08 is on 1.225 / 26.146

A very high ranking, should be clear because ...
5 engines analyzed ... and 4x another first "best" move.
Optimal for a test-set position and the ranking is high.

Note:
After the first Booot analyzes (engine 6) we have a A04 Position with 6 different best lines ... the first place in ranking.

:-)

Note: The current test-set is good but not perfect because 5 engines added only in FEOBOS. 8 others will be soon available. With more engines the test-set will be better because the ranking is more exactly.

Ah, here the game Komodo have no chance! Very perfect attacking game by Stockfish.

[pgn][Event "FEOBOS, 40/10, p=off, ht512, i7 4GHz"]
[Site "Trier"]
[Date "2017.06.16"]
[Round "217.1"]
[White "ASMFishW 20170522 BMI2 x64"]
[Black "Komodo 11.01 x64"]
[Result "1-0"]
[ECO "B08"]
[PlyCount "87"]
[EventDate "2017.??.??"]

1. d4 Nf6 2. Nf3 g6 3. Nc3 Bg7 4. e4 d6 5. Be3 a6 6. a4 O-O 7. Be2 b6 8. Nd2 {
[%eval 14,26] [%emt 0:00:52]} Bb7 {[%eval 3,22] [%emt 0:00:13]} 9. h4 {[%eval
4,26] [%emt 0:00:19] (O-O)} h5 {[%eval -14,23] [%emt 0:00:34] (c5)} 10. g4 {
[%eval 96,24] [%emt 0:00:07] (O-O)} hxg4 {[%eval 63,22] [%emt 0:00:14]} 11.
Bxg4 {[%eval 129,25] [%emt 0:00:16]} c5 {[%eval 65,24] [%emt 0:00:28]} 12. d5 {
[%eval 173,23] [%emt 0:00:06]} Nbd7 {[%eval 94,24] [%emt 0:00:40]} 13. h5 {
[%eval 174,26] [%emt 0:00:15]} Ne5 {[%eval 133,24] [%emt 0:00:38]} 14. Be2 {
[%eval 210,24] [%emt 0:00:05]} Re8 {[%eval 122,23] [%emt 0:00:15] (gxh5)} 15.
f4 {[%eval 262,23] [%emt 0:00:08]} Ned7 {[%eval 123,22] [%emt 0:00:05]} 16.
hxg6 {[%eval 299,24] [%emt 0:00:08] (h6)} fxg6 {[%eval 109,21] [%emt 0:00:05]}
17. Bg4 {[%eval 331,23] [%emt 0:00:05]} Nxg4 {[%eval 177,24] [%emt 0:00:44]
(e6)} 18. Qxg4 {[%eval 354,24] [%emt 0:00:07]} e6 {[%eval 272,23] [%emt 0:00:
11]} 19. Qxg6 {[%eval 408,26] [%emt 0:00:16] (dxe6)} exd5 {[%eval 315,21]
[%emt 0:00:09]} 20. f5 {[%eval 437,25] [%emt 0:00:03]} Nf8 {[%eval 406,22]
[%emt 0:00:18] (d4)} 21. Qg4 {[%eval 604,23] [%emt 0:00:11]} d4 {[%eval 442,24]
[%emt 0:00:21]} 22. Bh6 {[%eval 626,25] [%emt 0:00:15] (Tg1)} Qf6 {[%eval 472,
22] [%emt 0:00:18]} 23. Ne2 {[%eval 636,25] [%emt 0:00:06]} Re7 {[%eval 556,24]
[%emt 0:00:44] (Tad8)} 24. Bg5 {[%eval 752,23] [%emt 0:00:09]} Qe5 {[%eval 583,
22] [%emt 0:00:08] (Df7)} 25. Bf4 {[%eval 854,24] [%emt 0:00:10] (Lxe7)} Qf6 {
[%eval 714,23] [%emt 0:00:09]} 26. Rh6 {[%eval 869,22] [%emt 0:00:04]} Nh7 {
[%eval 816,26] [%emt 0:00:17]} 27. Rxf6 {[%eval 906,23] [%emt 0:00:10]} Nxf6 {
[%eval 894,26] [%emt 0:00:13]} 28. Qg6 {[%eval 919,24] [%emt 0:00:09]} Bxe4 {
[%eval 902,23] [%emt 0:00:07]} 29. Bh6 {[%eval 954,25] [%emt 0:00:12]} Bxc2 {
[%eval 960,24] [%emt 0:00:19] (Lxf5)} 30. Bxg7 {[%eval 1013,25] [%emt 0:00:28]}
Rxg7 {[%eval 1038,25] [%emt 0:00:18]} 31. Qxf6 {[%eval 1025,23] [%emt 0:00:05]}
Re8 {[%eval 1063,23] [%emt 0:00:12] (Tf8)} 32. Kf2 {[%eval 1215,26] [%emt 0:00:
38]} Bxf5 {[%eval 1090,23] [%emt 0:00:11]} 33. Nf4 {[%eval 1326,24] [%emt 0:00:
40]} Bh7 {[%eval 1188,23] [%emt 0:00:23] (Lg6)} 34. Ne6 {[%eval 12797,27]
[%emt 0:00:12]} Rg6 {[%eval 1250,26] [%emt 0:00:30]} 35. Rg1 {[%eval 12831,28]
[%emt 0:00:05]} Rxg1 {[%eval 1717,22] [%emt 0:00:06]} 36. Kxg1 {[%eval 32726,
34] [%emt 0:00:29]} Rxe6 {[%eval 1800,22] [%emt 0:00:03]} 37. Qxe6+ {[%eval
32746,34] [%emt 0:00:35]} Kf8 {[%eval 1843,20] [%emt 0:00:12] (Kg7)} 38. Qf6+ {
[%eval 32748,47] [%emt 0:00:27] (Dh6+)} Kg8 {[%eval 25000,22] [%emt 0:00:05]
(Ke8)} 39. Nc4 {[%eval 32756,72] [%emt 0:00:28] (Sf3)} d3 {[%eval 32758,29]
[%emt 0:00:00]} 40. Nxd6 {[%eval 32758,78] [%emt 0:00:12]} d2 {[%eval 32760,99]
[%emt 0:00:00]} 41. Qf7+ {[%eval 32760,94] [%emt 0:00:18]} Kh8 {[%eval 32762,5]
[%emt 0:00:00]} 42. Qf8+ {[%eval 32762,97] [%emt 0:00:04]} Bg8 {[%eval 32764,5]
[%emt 0:00:00]} 43. Nf7+ {[%eval 32764,127] [%emt 0:00:28]} Kh7 {[%eval 32766,
5] [%emt 0:00:00]} 44. Qh6# {[%eval 32766,127] [%emt 0:00:00]} 1-0
[/pgn]

Best
Frank
Nice Frank, as always with an early g4!
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Shortest, Komodo lost in 44, B08 on 1.225 (ranking)!

Post by Frank Quisinsky »

Bram,

oh yes!
Honestly, not often looking in B08 opening systems. For my book only. I will try it out in self play and will read a bit about it.

Thanks!

Best
Frank
carldaman
Posts: 2283
Joined: Sat Jun 02, 2012 2:13 am

Re: Shortest, Komodo lost in 44, B08 on 1.225 (ranking)!

Post by carldaman »

Frank Quisinsky wrote:Bram,

oh yes!
Honestly, not often looking in B08 opening systems. For my book only. I will try it out in self play and will read a bit about it.

Thanks!

Best
Frank
Hi Frank,

Would be possible to make all these (newer?) test games available for download? I am assuming they are not already part of your existing big database download.

Thanks,
CL
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Shortest, Komodo lost in 44, B08 on 1.225 (ranking)!

Post by Frank Quisinsky »

Hi Carl,

nice to read you!

I will do that after each 200 new games ... if I have B00-B99 complete
and after the next 200 games ... if I have C00-C99 complete.

Need a while, only one machine is still running for that match. Need my hardware power for the FEOBOS book project.

Today I created a little Dia-Show ...
If you have interest for looking ...

http://www.amateurschach.de/main/_new-opening-book.htm

Best
Frank
carldaman
Posts: 2283
Joined: Sat Jun 02, 2012 2:13 am

Re: Shortest, Komodo lost in 44, B08 on 1.225 (ranking)!

Post by carldaman »

Frank Quisinsky wrote:Hi Carl,

nice to read you!

I will do that after each 200 new games ... if I have B00-B99 complete
and after the next 200 games ... if I have C00-C99 complete.

Need a while, only one machine is still running for that match. Need my hardware power for the FEOBOS book project.

Today I created a little Dia-Show ...
If you have interest for looking ...

http://www.amateurschach.de/main/_new-opening-book.htm

Best
Frank
Thanks, Frank - whenever works for you, of course.
Nice to see your efforts continuing. :)
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Shortest, Komodo lost in 44, B08 on 1.225 (ranking)!

Post by Frank Quisinsky »

Hi Carl,

but not in eng-eng Play / Rating list or such material. I have enough from it, very boring for myself.

Only for Wasp testing!!

What I do here is to test a prototyp of FEBOBS Test-Set. End of the FEOBOS project with more engine analysis the final version of the test-set can be created. I will test a bit for looking how I can do that and where I have bad points in my FEOBOS database.

So I saw that for A31, E57 I need more material ... material without move transposition for FEOBOS database (often A31, E57 goes in move transpostions). End of the year ... all 13 engines analyzed the FEOBOS end positions I will make an database update (new games from 2017 ... and fixed database bad points).

So during this time I am searching and looking. Furthermore, we need a test for the Excel workbook. We are working hard to added interesting test-set options. Today we release version 61 (Topic so far ... queens on board or not ... more interesting for a test-set). Excel table need an option for it ... now with version 61 available in worksheet "eval-out". Each day a lot to do ... since weeks we are working the day and often the night on it!

Nothing to do with Stockfish vs. Komodo but with Stockfish vs. Komodo I can create a good alpha test.

Clear that others have fun with it too ... thats good.

In my interest is to test engines a bit more professional with postions engines gave us, not with position we gave the engines. We are not strong enough for it.

Time for FEOBOS!

Best
Frank
Frank Quisinsky
Posts: 6808
Joined: Wed Nov 18, 2009 7:16 pm
Location: Gutweiler, Germany
Full name: Frank Quisinsky

Re: Where is the high draw quote?

Post by Frank Quisinsky »

[PAIRINGS]
ASMFishW 20170522 BMI2 x64 - Komodo 11.01 x64
=1==1=1===111=========1====11=1=====11=====001===1
=1====111==1=1=======1=1===1===1=1=11=0===1====10=
==1==101===10===1==110===1======1==1=1===1=1=01===
=011========111======1==0=010111=11=1==10===01=1===1
1====1=========101=10==1===1===1===1=1=1=1=11111=111
=0=1====1==11=10==0======1=========01======1===11=
1=====01=1==011=======1==1=1==0=====1======1===10=
==0==1=1===11==1=1=XX=XX

Best
Frank

Answere:
PS: In all of the 500 test-position no 0,00 draw evals after 5 analyzed engines. And all are full balanced, which such a database I have here a lot of fun in visiting the games.

Ranking of positions can be see in Excel file.
Each of the 26.146 FEOBOS positions have an own ranking.

Unfortunately, no chance for Komodo 11.01 at the moment.
Stockfish without contempt ... not necessary with the database I am using.