ICGA's 2015 World Computer Chess Championship/Events

[IanO]

In my opinion, the TCEC is very close to a world championship, but not quite there yet.

Good:

The TCEC is attracting the actual top computer programs in the world today. The first missing engine is Rybka this year, and almost all other top engines were represented. By contrast, the top TWELVE computer programs were missing from the last WCCC. That alone shows the ICGA has lost touch with the needs of the community.

I also have no problem with the TCEC being an invitational, as long as the best programs are not excluded.

Automated play. There is absolutely no reason for participants to play on a physical board. Both UCI and WB protocols are established and reasonable to implement.

Decent hardware.

Good format. Multiple stages and long matches give a good mix of inclusivity and statistical rigor.

Excellent showmanship. It is hard to imagine how to improve the TCEC website. It is a joy to follow the matches and thought processes in real time.

Decent public relations. I have seen several articles covering the TCEC in both chess and mainstream journals.

Missing:

Own books. A championship should be the full effort of one opponent against the other, and that includes opening preparation. Engines should use their own books. Even better would be tuning for specific opponents, since opponent modeling is still an unsolved problem. Although problematic for the long format of the TCEC, it might be more workable for a shorter tournament. Having one stage be bookless this year was a good compromise.

(In general, it feels like bookmaking has stagnated since the end of Rybka and the rise of rating lists. It would be nice to reverse that trend. The computer chess community has a lot to offer the world of opening theory if they had a reason to focus on it.)

No adjudication. The TCEC game ending rules are OK to shorten a very lengthy format, but still have the potential for introducing error. A real championship should play to the bitter end.

Own endgame databases. Granted, the last season was an experiment, but I think a real championship should have the engines play with their preferred choice of endgame databases.

FIDE recognition. To be the World Championship, you must go through the political effort of wresting the title away from the ICGA. (Good luck with that; Levy is firmly entrenched.)

Once upon a time, I would have insisted on own hardware for a real WCCC, but the supercomputer and custom ASIC era has come to an end. There simply isn't the interest or funding to do that any more. Uniform high-end multiprocessing workstations are sufficient. The only real enhancement here is to have a computer dedicated to each opponent, with pondering.

[Modern Times]

Missing:

Own books. A championship should be the full effort of one opponent against the other, and that includes opening preparation. Engines should use their own books.

Everyone has different opinions on this, but I disagree on that. While respecting that bookmaking is an art and requires considerable skill, I hate own books being used. A good opening book can flatter a weaker engine, and I think that is wrong. I don't want to see that. I want to see the skills of the engine programmer shine through, with everyone using the same generic book.

[Harvey Williamson]

Missing:

Own books. A championship should be the full effort of one opponent against the other, and that includes opening preparation. Engines should use their own books.

Everyone has different opinions on this, but I disagree on that. While respecting that bookmaking is an art and requires considerable skill, I hate own books being used. A good opening book can flatter a weaker engine, and I think that is wrong. I don't want to see that. I want to see the skills of the engine programmer shine through, with everyone using the same generic book.

While I don't have a huge issue with generic books don't they make a tournament similar to what you guys on the ratings lists are doing?

[Milos]

You and Milos seem to not realize what 100 and 200 Elo points difference means. 200 means that in a direct match it's about 55% win, 40% draw, 5% loss.
Anyway, I did simulations, having 5 engines 200 Elo points weaker than Stockfish, with a total of 11 participating engines, 10 games each engine:

Without hardware and book variables: Stockfish has 60% to win WCCC.
With hardware and book variables: Stockfish has about 50% to win WCCC.

I have no clue what and how you simulated, but clearly your result is wrong.
I wrote a sim of my own, with the following assumptions:
11 participants with ELO ratings:
R, R-150, R-200, R-200, R-250, R-250, R-300, R-300, R-400, R-400, R-400
which as you can see is even less favorable for the favorite (with rating R) than your assumption.
And after the sim I got probability of favorite winning of roughly 90%.
Probability of the second one winning roughly 6%, others below 1.5%.

[Laskos]

You and Milos seem to not realize what 100 and 200 Elo points difference means. 200 means that in a direct match it's about 55% win, 40% draw, 5% loss.
Anyway, I did simulations, having 5 engines 200 Elo points weaker than Stockfish, with a total of 11 participating engines, 10 games each engine:

Without hardware and book variables: Stockfish has 60% to win WCCC.
With hardware and book variables: Stockfish has about 50% to win WCCC.

I have no clue what and how you simulated, but clearly your result is wrong.
I wrote a sim of my own, with the following assumptions:
11 participants with ELO ratings:
R, R-150, R-200, R-200, R-250, R-250, R-300, R-300, R-400, R-400, R-400
which as you can see is even less favorable for the favorite (with rating R) than your assumption.
And after the sim I got probability of favorite winning of roughly 90%.
Probability of the second one winning roughly 6%, others below 1.5%.

You sure got something wrong. I use drawelo 200 for draw ratios. Also I have 3 bopo engines (rating 1000 lower than SF, as it often happens in WCCC), but that shouldn't matter much. My result with your ratings for first 8 engines (the rest are bopo) is 58%.

[bob]

Now personally, I don't really care who calls themselves world champion and who calls themselves number one in the world

Maybe, just maybe, you will change your mind when your creation becomes as strong as SF and Komodo.

- and I'm utterly sick of people getting into heated emotional arguments over it. Get a life people.

For many many people this (the strongest engine) was (and still is) the spark that set the CC fire in motion.

Would you please stop making that bullshit statement? This "fire" was the direct result of violating a rule. Nothing more, nothing less...

[bob]

In my opinion, the TCEC is very close to a world championship, but not quite there yet.

Good:

The TCEC is attracting the actual top computer programs in the world today. The first missing engine is Rybka this year, and almost all other top engines were represented. By contrast, the top TWELVE computer programs were missing from the last WCCC. That alone shows the ICGA has lost touch with the needs of the community.

I also have no problem with the TCEC being an invitational, as long as the best programs are not excluded.

Automated play. There is absolutely no reason for participants to play on a physical board. Both UCI and WB protocols are established and reasonable to implement.

Decent hardware.

Good format. Multiple stages and long matches give a good mix of inclusivity and statistical rigor.

Excellent showmanship. It is hard to imagine how to improve the TCEC website. It is a joy to follow the matches and thought processes in real time.

Decent public relations. I have seen several articles covering the TCEC in both chess and mainstream journals.

Missing:

Own books. A championship should be the full effort of one opponent against the other, and that includes opening preparation. Engines should use their own books. Even better would be tuning for specific opponents, since opponent modeling is still an unsolved problem. Although problematic for the long format of the TCEC, it might be more workable for a shorter tournament. Having one stage be bookless this year was a good compromise.

(In general, it feels like bookmaking has stagnated since the end of Rybka and the rise of rating lists. It would be nice to reverse that trend. The computer chess community has a lot to offer the world of opening theory if they had a reason to focus on it.)

No adjudication. The TCEC game ending rules are OK to shorten a very lengthy format, but still have the potential for introducing error. A real championship should play to the bitter end.

Own endgame databases. Granted, the last season was an experiment, but I think a real championship should have the engines play with their preferred choice of endgame databases.

FIDE recognition. To be the World Championship, you must go through the political effort of wresting the title away from the ICGA. (Good luck with that; Levy is firmly entrenched.)

Once upon a time, I would have insisted on own hardware for a real WCCC, but the supercomputer and custom ASIC era has come to an end. There simply isn't the interest or funding to do that any more. Uniform high-end multiprocessing workstations are sufficient. The only real enhancement here is to have a computer dedicated to each opponent, with pondering.

Also a drawback: uniform platform.

[bob]

Missing:

Own books. A championship should be the full effort of one opponent against the other, and that includes opening preparation. Engines should use their own books.

Everyone has different opinions on this, but I disagree on that. While respecting that bookmaking is an art and requires considerable skill, I hate own books being used. A good opening book can flatter a weaker engine, and I think that is wrong. I don't want to see that. I want to see the skills of the engine programmer shine through, with everyone using the same generic book.

Then one has to ask why doesn't human tournament play go this way, as is done in one type of checkers, draw an opening position from a hat and the game starts there???

That's not exactly "chess" as most of us learned to play it, where opening preparation is just as important as middle game tactics and endgame knowledge.

[Milos]

You and Milos seem to not realize what 100 and 200 Elo points difference means. 200 means that in a direct match it's about 55% win, 40% draw, 5% loss.
Anyway, I did simulations, having 5 engines 200 Elo points weaker than Stockfish, with a total of 11 participating engines, 10 games each engine:

Without hardware and book variables: Stockfish has 60% to win WCCC.
With hardware and book variables: Stockfish has about 50% to win WCCC.

I have no clue what and how you simulated, but clearly your result is wrong.
I wrote a sim of my own, with the following assumptions:
11 participants with ELO ratings:
R, R-150, R-200, R-200, R-250, R-250, R-300, R-300, R-400, R-400, R-400
which as you can see is even less favorable for the favorite (with rating R) than your assumption.
And after the sim I got probability of favorite winning of roughly 90%.
Probability of the second one winning roughly 6%, others below 1.5%.

You sure got something wrong. I use drawelo 200 for draw ratios. Also I have 3 bopo engines (rating 1000 lower than SF, as it often happens in WCCC), but that shouldn't matter much. My result with your ratings for first 8 engines (the rest are bopo) is 58%.

I don't use drawelo 200 instead I use my own draw percentages for each Elo difference (these are much more realistic values for LTC matches):
0Elo - 64%, 50Elo - 60%, 100Elo - 52%, 150Elo - 45%, 200Elo - 38%, 250Elo - 34.3%, 300Elo - 28.2% and 400Elo - 17.6%
Here is the map of win/draw/loss probabilities for all 55 possible pairings (first against second, then third, etc.) between 11 engines with upper rating differences.

Code: Select all

    0.4780    0.4500    0.0720
    0.5700    0.3800    0.0500
    0.5700    0.3800    0.0500
    0.6370    0.3430    0.0200
    0.6370    0.3430    0.0200
    0.7080    0.2820    0.0100
    0.7080    0.2820    0.0100
    0.8210    0.1760    0.0030
    0.8210    0.1760    0.0030
    0.8210    0.1760    0.0030
    0.2710    0.6000    0.1290
    0.2710    0.6000    0.1290
    0.3800    0.5200    0.1000
    0.3800    0.5200    0.1000
    0.4780    0.4500    0.0720
    0.4780    0.4500    0.0720
    0.6370    0.3430    0.0200
    0.6370    0.3430    0.0200
    0.6370    0.3430    0.0200
    0.1800    0.6400    0.1800
    0.2710    0.6000    0.1290
    0.2710    0.6000    0.1290
    0.3800    0.5200    0.1000
    0.3800    0.5200    0.1000
    0.5700    0.3800    0.0500
    0.5700    0.3800    0.0500
    0.5700    0.3800    0.0500
    0.2710    0.6000    0.1290
    0.2710    0.6000    0.1290
    0.3800    0.5200    0.1000
    0.3800    0.5200    0.1000
    0.5700    0.3800    0.0500
    0.5700    0.3800    0.0500
    0.5700    0.3800    0.0500
    0.1800    0.6400    0.1800
    0.2710    0.6000    0.1290
    0.2710    0.6000    0.1290
    0.4780    0.4500    0.0720
    0.4780    0.4500    0.0720
    0.4780    0.4500    0.0720
    0.2710    0.6000    0.1290
    0.2710    0.6000    0.1290
    0.4780    0.4500    0.0720
    0.4780    0.4500    0.0720
    0.4780    0.4500    0.0720
    0.1800    0.6400    0.1800
    0.3800    0.5200    0.1000
    0.3800    0.5200    0.1000
    0.3800    0.5200    0.1000
    0.3800    0.5200    0.1000
    0.3800    0.5200    0.1000
    0.3800    0.5200    0.1000
    0.1800    0.6400    0.1800
    0.1800    0.6400    0.1800
    0.1800    0.6400    0.1800

I don't know how you do the sim, but mine is pretty straight forward and certainly correct. Based on the random variable I define outcome of each of the 55 games and calculate the points.
You must be doing something wrong. It is quite obvious since probabilities you get are intuitively totally off.

[Laskos]

Fine, then we disagree on simulations, and on intuition too. Sure you do something wrong with your outlandish 90%.