komodo, stockfish etc

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

komodo, stockfish etc

Post by MikeB »

this was a gauntlet of three Craftys versus an assortment of engines - this time control was base 6.6 sec inc .11 seconds ( adjusted to match Bob's hardware where he tests with base 10 sec inc .17)

No GUI here, this uses c shell scripts and few command line apps written by Bob that manage engine vs engines matches with the lowest possible overhead- just set a few parameters and type submit and it's off and running - Bob has it setup so that that he can automatically feed various servers that were waiting for work...

R

Code: Select all

ank Name                       Elo    +    - games score oppo. draws 
   1 Stockfish 7 64 POPCNT     3254   20   20  3000   95%  2825    8% 
   2 Stockfish 120316 64 POPC  3240   19   19  3000   94%  2825    8% 
   3 Komodo 9.42 64-bit        3212   18   18  3000   93%  2825    9% 
   4 Komodo 9.3 64-bit         3184   17   17  3000   92%  2825   11% 
   5 Senpai 1.0                2958   11   11  3000   72%  2825   23% 
   6 Stockfish  160226 64bit   2945   11   11  3000   71%  2825   26% 
   7 Texel 1.05 64-bit         2875   11   11  3000   61%  2825   24% 
   8 Hakkapeliitta 3.0         2856   11   11  3000   59%  2825   23% 
   9 Crafty-25.1b              2836    6    6 12000   32%  2984   25% 
  10 Crafty-25.1a              2824    6    6 12000   30%  2986   25% 
  11 Crafty-25.0               2816    6    6 12000   29%  2987   24% 
same engines at much longer time control of base 1 min/ inc 1 sec

Code: Select all

Rank Name                       Elo    +    - games score oppo. draws 
   1 Komodo 9.42 64-bit        3306  107  107   150   97%  2803    5% 
   2 Komodo 9.3 64-bit         3285  102  102   150   96%  2803    4% 
   3 Stockfish 120316 64 POPC  3244   90   90   150   96%  2803    7% 
   4 Stockfish 7 64 POPCNT     3181   77   77   150   94%  2803   12% 
   5 Stockfish  160226 64bit   2971   52   52   150   78%  2803   26% 
   6 Texel 1.05 64-bit         2890   47   47   150   68%  2803   33% 
   7 Senpai 1.0                2877   47   47   150   66%  2803   33% 
   8 Hakkapeliitta 3.0         2837   47   47   150   60%  2803   33% 
   9 Crafty-25.1a              2814   26   26   600   30%  2982   33% 
  10 Crafty-25.1b              2809   27   27   600   30%  2983   27% 
  11 Crafty-25.0               2786   27   27   600   26%  2986   29% 
just saying it's interesting....will try to post a link to video to demonstrate
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: komodo, stockfish etc

Post by MikeB »

A 4 minute video demonstrating Bob's custom scripts and apps to run his GUI less crafty testing.

I have modified his scripts to run round robin(RR) matches and to show progress throughout the match. In this video , it was a 4 round RR using two test positions , each side playing with black and white. A very short micro time was chosen test to demo the process. In this case it was 1 sec base/ .1 sec inc. There were 11 engines, playing 4 games each with the other engines. Each engine played 40 games in total. The RR, 220 games in total, was completed in less than 5 minutes on a 12 core Mac Pro. The script has played 30,000 games consecutively without an error or crash.

GUI less RR Tourney
petero2
Posts: 736
Joined: Mon Apr 19, 2010 7:07 pm
Location: Sweden
Full name: Peter Osterlund

Re: komodo, stockfish etc

Post by petero2 »

MikeB wrote:this was a gauntlet of three Craftys versus an assortment of engines - this time control was base 6.6 sec inc .11 seconds ( adjusted to match Bob's hardware where he tests with base 10 sec inc .17)

No GUI here, this uses c shell scripts and few command line apps written by Bob that manage engine vs engines matches with the lowest possible overhead- just set a few parameters and type submit and it's off and running - Bob has it setup so that that he can automatically feed various servers that were waiting for work...

Code: Select all

Rank Name                       Elo    +    - games score oppo. draws 
   1 Stockfish 7 64 POPCNT     3254   20   20  3000   95%  2825    8% 
   2 Stockfish 120316 64 POPC  3240   19   19  3000   94%  2825    8% 
   3 Komodo 9.42 64-bit        3212   18   18  3000   93%  2825    9% 
   4 Komodo 9.3 64-bit         3184   17   17  3000   92%  2825   11% 
   5 Senpai 1.0                2958   11   11  3000   72%  2825   23% 
   6 Stockfish  160226 64bit   2945   11   11  3000   71%  2825   26% 
   7 Texel 1.05 64-bit         2875   11   11  3000   61%  2825   24% 
   8 Hakkapeliitta 3.0         2856   11   11  3000   59%  2825   23% 
   9 Crafty-25.1b              2836    6    6 12000   32%  2984   25% 
  10 Crafty-25.1a              2824    6    6 12000   30%  2986   25% 
  11 Crafty-25.0               2816    6    6 12000   29%  2987   24% 
This looks very strange. Why is stockfish 160226 about 300 elo weaker than stockfish 7?
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: komodo, stockfish etc

Post by MikeB »

petero2 wrote:
MikeB wrote:this was a gauntlet of three Craftys versus an assortment of engines - this time control was base 6.6 sec inc .11 seconds ( adjusted to match Bob's hardware where he tests with base 10 sec inc .17)

No GUI here, this uses c shell scripts and few command line apps written by Bob that manage engine vs engines matches with the lowest possible overhead- just set a few parameters and type submit and it's off and running - Bob has it setup so that that he can automatically feed various servers that were waiting for work...

Code: Select all

Rank Name                       Elo    +    - games score oppo. draws 
   1 Stockfish 7 64 POPCNT     3254   20   20  3000   95%  2825    8% 
   2 Stockfish 120316 64 POPC  3240   19   19  3000   94%  2825    8% 
   3 Komodo 9.42 64-bit        3212   18   18  3000   93%  2825    9% 
   4 Komodo 9.3 64-bit         3184   17   17  3000   92%  2825   11% 
   5 Senpai 1.0                2958   11   11  3000   72%  2825   23% 
   6 Stockfish  160226 64bit   2945   11   11  3000   71%  2825   26% 
   7 Texel 1.05 64-bit         2875   11   11  3000   61%  2825   24% 
   8 Hakkapeliitta 3.0         2856   11   11  3000   59%  2825   23% 
   9 Crafty-25.1b              2836    6    6 12000   32%  2984   25% 
  10 Crafty-25.1a              2824    6    6 12000   30%  2986   25% 
  11 Crafty-25.0               2816    6    6 12000   29%  2987   24% 
This looks very strange. Why is stockfish 160226 about 300 elo weaker than stockfish 7?
stockfish 2.3.1
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: komodo, stockfish etc

Post by MikeB »

Get real time updates on marathon 13 engine , 60,000 game per engine, GUI less Round Robin here:

Realtime updates every 5 minutes.

background:

Summary: 13 engines play each other 5000 times, with alternating colors, with a one of kind 2500 posiiton book crafted from the top (+3200) CCRL games which were decisive. Estimated completion time 2 weeks.

Objective: To set a high end baseline for future crafty engine testing.

more info here:
http://www.talkchess.com/forum/viewtopi ... 938#665938
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: komodo, stockfish etc

Post by MikeB »

MikeB wrote:Get real time updates on marathon 13 engine , 60,000 game per engine, GUI less Round Robin here:

Realtime updates every 5 minutes.

background:

Summary: 13 engines play each other 5000 times, with alternating colors, with a one of kind 2500 posiiton book crafted from the top (+3200) CCRL games which were decisive. Estimated completion time 2 weeks.

Objective: To set a high end baseline for future crafty engine testing.

more info here:
http://www.talkchess.com/forum/viewtopi ... 938#665938
MikeB wrote:Get real time updates on marathon 13 engine , 60,000 game per engine, GUI less Round Robin here:

Realtime updates every 5 minutes.

background:

Summary: 13 engines play each other 5000 times, with alternating colors, with a one of kind 2500 posiiton book crafted from the top (+3200) CCRL games which were decisive. Estimated completion time 2 weeks.

Objective: To set a high end baseline for future crafty engine testing.

more info here:
http://www.talkchess.com/forum/viewtopi ... 938#665938
Phase 2A has started, a 7800 game run that will take about 6 hours - same link as before for realtime updates. Also added a link that will just show the Phase 2A results.

ALL Realtime updates every 5 minutes.

Phase 2A Only - Realtime updates every 5 minutes.
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Phase 2B

Post by MikeB »

Phase 2B will start soon, a 15600 game run that will take about 15 hours - same link as before for realtime updates. Also included a link that will just show the Phase 2B results.

ALL Games Realtime updates every 5 minutes

Phase 2B Games Only - Realtime updates every 5 minutes

It should start within the next 15 minutes - check back often.
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Phase 2C

Post by MikeB »

Phase 2C will start soon, another 15600 game run that will take about 15 hours - same link as before for realtime updates. Also included a link that will just show the Phase 2B results.

ALL Games Realtime updates every 5 minutes

Phase 2C Games Only - Realtime updates every 5 minutes

It should start within the next 15 minutes - check back often.
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Phase 3

Post by MikeB »

Phase 3 will start soon, another 15600 game run that will take about 15 hours - same link as before for realtime updates. Also included a link that will just show the Phase 3 results.

ALL Games Realtime updates every 5 minutes

Phase 3 Games Only - Realtime updates every 5 minutes

It should start within the next 15 minutes - check back often.