ICGA's 2015 World Computer Chess Championship/Events

[Milos]

Missing:

Own books. A championship should be the full effort of one opponent against the other, and that includes opening preparation. Engines should use their own books. Even better would be tuning for specific opponents, since opponent modeling is still an unsolved problem. Although problematic for the long format of the TCEC, it might be more workable for a shorter tournament. Having one stage be bookless this year was a good compromise.

I won't comment books, but my personal opinion is that books have nothing to do with engine programming, and contrary to what ICGA (which claims to be the competition of programmers) is doing, TCEC should actually be the competition of best engine programming.

No adjudication. The TCEC game ending rules are OK to shorten a very lengthy format, but still have the potential for introducing error. A real championship should play to the bitter end.

This is pretty unreasonable. Why in human chess offering draws and resigning is allowed while in computer chess you would prevent it???
In all the games during all TCEC seasons there might have been 2 or 3 games in total where results was wrongly adjudicated, a percentage much lower than in case of human chess.

Own endgame databases. Granted, the last season was an experiment, but I think a real championship should have the engines play with their preferred choice of endgame databases.

TCEC already incorporates all existing endgame databases (except Robobases which no engine used), so this point is moot.

FIDE recognition. To be the World Championship, you must go through the political effort of wresting the title away from the ICGA. (Good luck with that; Levy is firmly entrenched.)

WCCC has nothing to do with FIDE, and never had (except for that one and only case when it had the same venue as FIDE Olympiad in Torino). Levy basically bribed someone from Ilyumzhinov administration 15 years ago to accept ICGA as FIDE affiliated organization (just one out of huge bunch).
Basically any organization that pays some money to Ilyumzhinov can become FIDE affiliated, so no big deal really.

[Milos]

Fine, then we disagree on simulations, and on intuition too. Sure you do something wrong with your outlandish 90%.

Well, I provide you details how I did the simulation (you can easily repeat it or find errors if they exist as you claim). Instead, you as always show only your imaginary numbers.
Result of most of your sims anyway had never had any sense, so I believe you are just pulling ppl's legs.

[Modern Times]

Then one has to ask why doesn't human tournament play go this way, as is done in one type of checkers, draw an opening position from a hat and the game starts there???

That's not exactly "chess" as most of us learned to play it, where opening preparation is just as important as middle game tactics and endgame knowledge.

Human chess and computer chess aren't totally comparable, and what might be good or possible for one may not be good or possible for the other. For example it is completely possible in computer chess to play without any book at all, and have the engine think from move 1. In human chess that is not possible, you can't turn that part of the brain off that has learned opening theory. So perhaps we should stop trying to compare the two.

[michiguel]

In my opinion, the TCEC is very close to a world championship, but not quite there yet.

Good:

The TCEC is attracting the actual top computer programs in the world today. The first missing engine is Rybka this year, and almost all other top engines were represented. By contrast, the top TWELVE computer programs were missing from the last WCCC. That alone shows the ICGA has lost touch with the needs of the community.

I also have no problem with the TCEC being an invitational, as long as the best programs are not excluded.

This is major issue that stops it for being a true WC. A true WC cannot be only by invitation (or selected to participate). There should be an avenue for anybody to have a shot, even if there is only one spot to qualify.

But I am not suggesting that. TCEC should keep being like it is. Being the most prestigious competition is not that bad. It does not have to be a WC. Some sport do not even have a WC!

Miguel

Automated play. There is absolutely no reason for participants to play on a physical board. Both UCI and WB protocols are established and reasonable to implement.

Decent hardware.

Good format. Multiple stages and long matches give a good mix of inclusivity and statistical rigor.

Excellent showmanship. It is hard to imagine how to improve the TCEC website. It is a joy to follow the matches and thought processes in real time.

Decent public relations. I have seen several articles covering the TCEC in both chess and mainstream journals.

Missing:

Own books. A championship should be the full effort of one opponent against the other, and that includes opening preparation. Engines should use their own books. Even better would be tuning for specific opponents, since opponent modeling is still an unsolved problem. Although problematic for the long format of the TCEC, it might be more workable for a shorter tournament. Having one stage be bookless this year was a good compromise.

(In general, it feels like bookmaking has stagnated since the end of Rybka and the rise of rating lists. It would be nice to reverse that trend. The computer chess community has a lot to offer the world of opening theory if they had a reason to focus on it.)

No adjudication. The TCEC game ending rules are OK to shorten a very lengthy format, but still have the potential for introducing error. A real championship should play to the bitter end.

Own endgame databases. Granted, the last season was an experiment, but I think a real championship should have the engines play with their preferred choice of endgame databases.

FIDE recognition. To be the World Championship, you must go through the political effort of wresting the title away from the ICGA. (Good luck with that; Levy is firmly entrenched.)

Once upon a time, I would have insisted on own hardware for a real WCCC, but the supercomputer and custom ASIC era has come to an end. There simply isn't the interest or funding to do that any more. Uniform high-end multiprocessing workstations are sufficient. The only real enhancement here is to have a computer dedicated to each opponent, with pondering.

[petero2]

I wrote a sim of my own, with the following assumptions:
11 participants with ELO ratings:
R, R-150, R-200, R-200, R-250, R-250, R-300, R-300, R-400, R-400, R-400
And after the sim I got probability of favorite winning of roughly 90%.
Probability of the second one winning roughly 6%, others below 1.5%.

I don't use drawelo 200 instead I use my own draw percentages for each Elo difference (these are much more realistic values for LTC matches):
0Elo - 64%, 50Elo - 60%, 100Elo - 52%, 150Elo - 45%, 200Elo - 38%, 250Elo - 34.3%, 300Elo - 28.2% and 400Elo - 17.6%

Using these parameters and ignoring the white advantage since I don't know how colors are assigned in this tournament, I got after 1e8 simulated tournaments:

Code: Select all

eng   elo  win prob
  1     0  0.801288
  2  -150  0.0576597
  3  -200  0.0154924
  4  -200  0.0155108
  5  -250  0.00279631
  6  -250  0.00279635
  7  -300  0.00041849
  8  -300  0.00042163
  9  -400  3.9e-06
 10  -400  4.1e-06
 11  -400  3.61e-06
ties:      0.103605

Note that I have not implemented the tie-break rules, so I don't know how those 10.4% ties would be distributed among the participants.

[Laskos]

I wrote a sim of my own, with the following assumptions:
11 participants with ELO ratings:
R, R-150, R-200, R-200, R-250, R-250, R-300, R-300, R-400, R-400, R-400
And after the sim I got probability of favorite winning of roughly 90%.
Probability of the second one winning roughly 6%, others below 1.5%.

I don't use drawelo 200 instead I use my own draw percentages for each Elo difference (these are much more realistic values for LTC matches):
0Elo - 64%, 50Elo - 60%, 100Elo - 52%, 150Elo - 45%, 200Elo - 38%, 250Elo - 34.3%, 300Elo - 28.2% and 400Elo - 17.6%

Using these parameters and ignoring the white advantage since I don't know how colors are assigned in this tournament, I got after 1e8 simulated tournaments:

Code: Select all

eng   elo  win prob
  1     0  0.801288
  2  -150  0.0576597
  3  -200  0.0154924
  4  -200  0.0155108
  5  -250  0.00279631
  6  -250  0.00279635
  7  -300  0.00041849
  8  -300  0.00042163
  9  -400  3.9e-06
 10  -400  4.1e-06
 11  -400  3.61e-06
ties:      0.103605

Note that I have not implemented the tie-break rules, so I don't know how those 10.4% ties would be distributed among the participants.

In the case of a tie, I applied random proportional probability (50% to each in case of 2, 33.3..% in case of 3 and so on). What is the draw model?

[petero2]

I wrote a sim of my own, with the following assumptions:
11 participants with ELO ratings:
R, R-150, R-200, R-200, R-250, R-250, R-300, R-300, R-400, R-400, R-400
And after the sim I got probability of favorite winning of roughly 90%.
Probability of the second one winning roughly 6%, others below 1.5%.

I don't use drawelo 200 instead I use my own draw percentages for each Elo difference (these are much more realistic values for LTC matches):
0Elo - 64%, 50Elo - 60%, 100Elo - 52%, 150Elo - 45%, 200Elo - 38%, 250Elo - 34.3%, 300Elo - 28.2% and 400Elo - 17.6%

Using these parameters and ignoring the white advantage since I don't know how colors are assigned in this tournament, I got after 1e8 simulated tournaments:

Code: Select all

eng   elo  win prob
  1     0  0.801288
  2  -150  0.0576597
  3  -200  0.0154924
  4  -200  0.0155108
  5  -250  0.00279631
  6  -250  0.00279635
  7  -300  0.00041849
  8  -300  0.00042163
  9  -400  3.9e-06
 10  -400  4.1e-06
 11  -400  3.61e-06
ties:      0.103605

Note that I have not implemented the tie-break rules, so I don't know how those 10.4% ties would be distributed among the participants.

In the case of a tie, I applied random proportional probability (50% to each in case of 2, 33.3..% in case of 3 and so on). What is the draw model?

My draw model is defined by the data points given by Milos, with linear interpolation between points. Although for the elo differences in this example, no interpolation is needed.

I implemented the first tie-break rule: If exactly 2 participants are tied for first place they play two additional games. This gets the remaining ties down to 3.5%:

Code: Select all

eng   elo  win prob
  1     0  0.861052
  2  -150  0.063327
  3  -200  0.0168844
  4  -200  0.016892
  5  -250  0.0029813
  6  -250  0.00297884
  7  -300  0.00044592
  8  -300  0.00044474
  9  -400  4.43e-06
 10  -400  4.05e-06
 11  -400  4.28e-06
ties:3498130 0.0349813

I can try with your draw model too. Is it correct that you use these formulas from the bayeselo documentation:

Code: Select all

f(Delta) = 1 / (1 + 10^(Delta/400))
P(WhiteWins) = f(eloBlack - eloWhite - eloAdvantage + eloDraw)
P(BlackWins) = f(eloWhite - eloBlack + eloAdvantage + eloDraw)
P(Draw) = 1 - P(WhiteWins) - P(BlackWins)

with eloAdvantage = 0 and eloDraw = 200?

[Laskos]

I can try with your draw model too. Is it correct that you use these formulas from the bayeselo documentation:

Code: Select all

f(Delta) = 1 / (1 + 10^(Delta/400))
P(WhiteWins) = f(eloBlack - eloWhite - eloAdvantage + eloDraw)
P(BlackWins) = f(eloWhite - eloBlack + eloAdvantage + eloDraw)
P(Draw) = 1 - P(WhiteWins) - P(BlackWins)

with eloAdvantage = 0 and eloDraw = 200?

Yes, I don't keep colors, and drawelo=200.

[petero2]

I can try with your draw model too. Is it correct that you use these formulas from the bayeselo documentation:

Code: Select all

f(Delta) = 1 / (1 + 10^(Delta/400))
P(WhiteWins) = f(eloBlack - eloWhite - eloAdvantage + eloDraw)
P(BlackWins) = f(eloWhite - eloBlack + eloAdvantage + eloDraw)
P(Draw) = 1 - P(WhiteWins) - P(BlackWins)

with eloAdvantage = 0 and eloDraw = 200?

Yes, I don't keep colors, and drawelo=200.

In that case I get:

Code: Select all

eng   elo  win prob
  1     0  0.734052
  2  -150  0.0947824
  3  -200  0.0381066
  4  -200  0.038102
  5  -250  0.0134178
  6  -250  0.0133863
  7  -300  0.00408722
  8  -300  0.00408363
  9  -400  0.00024338
 10  -400  0.0002405
 11  -400  0.00024259
ties:      0.0592556

Note though that this draw model also distorts the rating scale.

Code: Select all

elo  s1      s2
  0  0.5000  0.5000
 20  0.5288  0.5210
 40  0.5573  0.5420
 60  0.5855  0.5629
 80  0.6131  0.5838
100  0.6401  0.6045
120  0.6661  0.6250
140  0.6912  0.6454
160  0.7153  0.6654
180  0.7381  0.6852
200  0.7597  0.7045
220  0.7801  0.7235
240  0.7992  0.7419
260  0.8171  0.7597
280  0.8337  0.7769
300  0.8490  0.7934
320  0.8632  0.8092
340  0.8762  0.8242
360  0.8882  0.8385
380  0.8991  0.8519
400  0.9091  0.8645

s1 is the win rate according to the standard elo model and s2 is the win rate according to the above draw model.

[Milos]

I wrote a sim of my own, with the following assumptions:
11 participants with ELO ratings:
R, R-150, R-200, R-200, R-250, R-250, R-300, R-300, R-400, R-400, R-400
And after the sim I got probability of favorite winning of roughly 90%.
Probability of the second one winning roughly 6%, others below 1.5%.

I don't use drawelo 200 instead I use my own draw percentages for each Elo difference (these are much more realistic values for LTC matches):
0Elo - 64%, 50Elo - 60%, 100Elo - 52%, 150Elo - 45%, 200Elo - 38%, 250Elo - 34.3%, 300Elo - 28.2% and 400Elo - 17.6%

Using these parameters and ignoring the white advantage since I don't know how colors are assigned in this tournament, I got after 1e8 simulated tournaments:

Code: Select all

eng   elo  win prob
  1     0  0.801288
  2  -150  0.0576597
  3  -200  0.0154924
  4  -200  0.0155108
  5  -250  0.00279631
  6  -250  0.00279635
  7  -300  0.00041849
  8  -300  0.00042163
  9  -400  3.9e-06
 10  -400  4.1e-06
 11  -400  3.61e-06
ties:      0.103605

Note that I have not implemented the tie-break rules, so I don't know how those 10.4% ties would be distributed among the participants.

In the case of a tie, I applied random proportional probability (50% to each in case of 2, 33.3..% in case of 3 and so on). What is the draw model?

My draw model is defined by the data points given by Milos, with linear interpolation between points. Although for the elo differences in this example, no interpolation is needed.

I implemented the first tie-break rule: If exactly 2 participants are tied for first place they play two additional games. This gets the remaining ties down to 3.5%:

Code: Select all

eng   elo  win prob
  1     0  0.861052
  2  -150  0.063327
  3  -200  0.0168844
  4  -200  0.016892
  5  -250  0.0029813
  6  -250  0.00297884
  7  -300  0.00044592
  8  -300  0.00044474
  9  -400  4.43e-06
 10  -400  4.05e-06
 11  -400  4.28e-06
ties:3498130 0.0349813

The remaining 3.5% is resolved by additional sudden death game.
You can assume 50% probability to play white and participant that plays white has additional 45Elo advantage (30Elo due to playing white and 15Elo due to 20% more time).

In my calculations and didn't use tie rules but instead I assumed that the percentage of tie points is distributed approximately as the percentage of non-tie points, i.e.

Code: Select all

win_prob_wo_tie*(1+1/(1-ties)*ties) = win_prob_wo_tie*1.11558
eng   elo  win prob
  1     0  0.801288  *1.11558 = 0.893900
  2  -150  0.0576597 *1.11558 = 0.064324
  3  -200  0.0154924 *1.11558 = 0.017283
  4  -200  0.0155108 *1.11558 = 0.017303
  5  -250  0.00279631*1.11558 = 0.003119
  6  -250  0.00279635*1.11558 = 0.003119
  7  -300  0.00041849*1.11558 = 0.000466
  8  -300  0.00042163*1.11558 = 0.000470

Considering that almost 90% of the remaining 3.5% ties is played by engine 1 and that it wins at least 75% of those, final percentage for engine 1 winning is 86.1+0.68*3.5=88.5% which is pretty close to the upper calculation I used to resolve ties.

Anyway, point is that even with these slightly unrealistic conditions unfavoring SF, SF would comfortably win ICGA with almost 90% probability!