March Madness

[tsmiller1]

In light of the recently completed NCAA Basketball Tournament (GO Blue Devils!), I have decided to hold my own little March Madness for chess engines. This will no doubt not be terribly rigorous engine testing, just something I think will be fun.

My plans are to hold a 64-engine G/60 tournament of match play between the top 64 ranked (as per CCRL) free or open-source engines. I'm going to add one of the Ippolit family engines to the mix, but otherwise use the ratings as per CCRL's 32-bit, 1-CPU 40/40 list for free and open-source engines.

Round 1 will be Best of 8. Round 2 will be Best of 12. Round 3 ("Sweet Sixteen") will be Best of 16. Quarterfinals ("Elite Eight") will be Best of 24. Semi-Finals ("Final Four") will be Best of 30, and the Finals will be Best of 40.

After all this is completed, I'm going to match the winner of the free engine championship with the best available commercial engine (currently Rybka 3) in a sort of David vs. Goliath match, Best of 50.

I plan to use the Arena Little Mainbook limited to 8 moves, along with the Nalimov 3-4-5 EGTB's. Arena interface.

Any thought or ideas before I get started?

[hgm]

I did not even know there were that many open-source engines! Would micro-Max be amongst those, or is it too much down on the list?

[tsmiller1]

I did not even know there were that many open-source engines! Would micro-Max be amongst those, or is it too much down on the list?

There are many more than just 64! It looks like micro-Max comes in at 168 in the CCRL list. I made a decision just to go with the top 64 free engines to sort of emulate the NCAA Championships.

Check out the CCRL list to which I refer: http://computerchess.org.uk/ccrl/4040.l ... ons_only=1

[hgm]

Micro-Max has a 168 in front of its name on that list, but I am not sure what that means, as there are many engines above it that do not have a number at all. But it is a list of all free engines. Only the red entries are free open-source engines. You mean you are going to play all those infinitessimally different Toga II versions against each other?

[tsmiller1]

Micro-Max has a 168 in front of its name on that list, but I am not sure what that means, as there are many engines above it that do not have a number at all. But it is a list of all free engines. Only the red entries are free open-source engines. You mean you are going to play all those infinitessimally different Toga II versions against each other?

Well, that particular CCRL list is a list of "best version only", which means that only the "best" of each family of engines is ranked. All the other versions are listed for the sake of comparison, but only the best version is ranked. For example, in the Fruit family, the best version is the Grapefruit 1.0 engine, which is currently ranked 5th. But you see listed the other members of the Fruit family (Toga, Cyclone, etc.)

This means that (according to the CCRL list, anyway), Micro-Max is the 168th highest rated unique engine. I think to be considered a unique engine it has to have an author and not be derived from other programs in a certain fashion.

The red engines are open-source, the green engines are free engines (which apparently are not necessarily open-source). I'm not a programmer (can't you tell?).

[Graham Banks]

In light of the recently completed NCAA Basketball Tournament (GO Blue Devils!), I have decided to hold my own little March Madness for chess engines. This will no doubt not be terribly rigorous engine testing, just something I think will be fun.

My plans are to hold a 64-engine G/60 tournament of match play between the top 64 ranked (as per CCRL) free or open-source engines. I'm going to add one of the Ippolit family engines to the mix, but otherwise use the ratings as per CCRL's 32-bit, 1-CPU 40/40 list for free and open-source engines.

Round 1 will be Best of 8. Round 2 will be Best of 12. Round 3 ("Sweet Sixteen") will be Best of 16. Quarterfinals ("Elite Eight") will be Best of 24. Semi-Finals ("Final Four") will be Best of 30, and the Finals will be Best of 40.

After all this is completed, I'm going to match the winner of the free engine championship with the best available commercial engine (currently Rybka 3) in a sort of David vs. Goliath match, Best of 50.

I plan to use the Arena Little Mainbook limited to 8 moves, along with the Nalimov 3-4-5 EGTB's. Arena interface.

Any thought or ideas before I get started?

Sounds good. If you're still enthusiastic afterwards, you could always run one for the next 64.

Cheers,
Graham.

[tsmiller1]

OK. I've got the brackets completed, as well as the first match under my belt. I decided to play the matches for Bracket A first, then Bracket B, etc., instead of simply playing 1 vs. 64, then 2 vs. 63, etc., to give the Brackets their own feel (like the real March Madness). Here's the link to my Google spreadsheet with the matchups, in case anyone is interested. I'll try to update it as much as I can as I complete the matches. The link is:

http://spreadsheets.google.com/ccc?key= ... aX2c&hl=en

As I expected the first battle, pitting Bracket A's No. 1 Seed Ippolit 0.080b3 vs No. 16 seed N2 0.1 was a no-contest, with Ippolit winning easily 5-0. I can post the games if anyone is interested.

[hgm]

The red engines are open-source, the green engines are free engines (which apparently are not necessarily open-source). I'm not a programmer (can't you tell?).

I don't think you have to be a programmer to understand what 'open source' means anymore than you have to be an economist to know what 'free' means.

But to put things right: you are not going to play the strongest 64 open-source engines, but the strongest 64 free engines according to the CCRL list. The number of open source engines on that list is smaller than 64, (although larger than I realized), so I was jusified in questioning what you were saying. If you say something completely different from what you mean, you should not be surprized if people misunderstand you...

It is also April, btw...

[Carlos777]

As I expected the first battle, pitting Bracket A's No. 1 Seed Ippolit 0.080b3 vs No. 16 seed N2 0.1 was a no-contest, with Ippolit winning easily 5-0. I can post the games if anyone is interested.

Nice idea, but why are you using older versions?

For example:

Ippolit 0.080b3 - Robbolito 0.09 (the latest single core version)
N2 v0.1 - 0.4
Danasah 4.24 - 4.37
Stockfish 1.6.3 - 1.7.1
Nanoszachy 3.7 - 3.8
Tornado 2.2 - 3.42a
Crafty 23.1 - 23.2
Arasan 10.0 - 11.7
Critter 0.52b - 0.60
Trace 1.36 - 1.37
ProDeo 1.2 - 1.6
Booot 4.15.0 - 4.15.1
etc.

Regards,
Carlos

[Sven]

The red engines are open-source, the green engines are free engines (which apparently are not necessarily open-source). I'm not a programmer (can't you tell?).

I don't think you have to be a programmer to understand what 'open source' means anymore than you have to be an economist to know what 'free' means.

But to put things right: you are not going to play the strongest 64 open-source engines, but the strongest 64 free engines according to the CCRL list. The number of open source engines on that list is smaller than 64, (although larger than I realized), so I was jusified in questioning what you were saying. If you say something completely different from what you mean, you should not be surprized if people misunderstand you...

It is also April, btw...

I think the very first posting was clear in this point (my bold added):

My plans are to hold a 64-engine G/60 tournament of match play between the top 64 ranked (as per CCRL) free or open-source engines. I'm going to add one of the Ippolit family engines to the mix, but otherwise use the ratings as per CCRL's 32-bit, 1-CPU 40/40 list for free and open-source engines.

Sven