ICGA's 2015 World Computer Chess Championship/Events

[Milos]

I got this with the Ordo model

Code: Select all

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 50.0
rating[0]=0
rating[1]=-150
rating[2]=-200
rating[3]=-200
rating[4]=-250
rating[5]=-250
rating[6]=-300
rating[7]=-300
rating[8]=-400
rating[9]=-400
rating[10]=-400

won    = 827663
shared = 89338
loss   = 82999
total  = 1000000
won outright % = 82.8
won shared   % = 8.9

It is including white advantage and assuming 64% draw rate for equal strength.

Miguel

White advantage of 50Elo is unrealistic. In reality it is rarely above 30Elo.
How does it resolve tie brakes?
You have 8.9% tie brakes which seams to be ok, but it is not clear weather you assume all them as won.

[Milos]

Code: Select all

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 50.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200

won    = 631963
shared = 175635
loss   = 192402
total  = 1000000
won outright % = 63.2
won shared   % = 17.6

Miguel

This also looks correct.
So even with only 100Elo advantage over second rated engine, which in reality should be more than 200Elo in the worst case, meaning this rating distribution is totally bogus and hypothetical, SF would have around 80% chance of winning ICGA.

@Kai, if you still don't see how your "sims" (if you even sim anything after all) are totally bogus (in these conditions you claim SF would not even have 50% chance of winning), no one can really help you.

[Laskos]

This also looks correct.
So even with only 100Elo advantage over second rated engine, which in reality should be more than 200Elo in the worst case, meaning this rating distribution is totally bogus and hypothetical, SF would have around 80% chance of winning ICGA.

@Kai, if you still don't see how your "sims" (if you even sim anything after all) are totally bogus (in these conditions you claim SF would not even have 50% chance of winning), no one can really help you.

It depends on draw model, with that used in bayeselo my 60% was pretty close. Also on variability on hardware and book, if you include 100 Elo for hardware and 50 for book, as I proposed when making 50% claim for SF, you will get 50% with drawelo 200. It only gets closer to your result with your flimsy "draw model".

[Adam Hair]

I got this with the Ordo model

Code: Select all

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 50.0
rating[0]=0
rating[1]=-150
rating[2]=-200
rating[3]=-200
rating[4]=-250
rating[5]=-250
rating[6]=-300
rating[7]=-300
rating[8]=-400
rating[9]=-400
rating[10]=-400

won    = 827663
shared = 89338
loss   = 82999
total  = 1000000
won outright % = 82.8
won shared   % = 8.9

It is including white advantage and assuming 64% draw rate for equal strength.

Miguel

White advantage of 50Elo is unrealistic. In reality it is rarely above 30Elo.

White advantage increases as the average strength of the opponents increases. I am not saying that 50 Elo is a good assumption under the assumed conditions, for I have not had time to follow what the 3 of you are doing. But, you should not just assume that the advantage should be under 30 Elo.

[hgm]

Everyone has different opinions on this, but I disagree on that. While respecting that bookmaking is an art and requires considerable skill, I hate own books being used. A good opening book can flatter a weaker engine, and I think that is wrong. I don't want to see that. I want to see the skills of the engine programmer shine through, with everyone using the same generic book.

Deeper search can also flatter an engine with weaker evaluation. So what if I prefer to have "the skill of the evaluation designer shine through"? Would it be better to have the WCCC run at fixed depth?

And fixed depth also flatter weaker engines that extend more. So should we just let the participents supply a DLL with an evaluation function, so that they could all compete using a standard search supplied by the organizers.

Actually you could also interpret the same fact as that "strong engines flatter weaker books", (or slower hardware, for that matter...), and that for that reason everyone in the WCCC should use the same engine!

As far as I am concerned all this is equally nonsensical. The WCCC is a contest for creating (as opposed to copying...) the AI that plays the best Chess. Opening theory is part of Chess. A tournament that arbitrarily excludes it is not a Chess championship, anymore than making the world's best tyres is Formula I racing. It could be called the WCCEB (Worl Championship Chess-Engine Building).

Especially because opening books can 'flatter weaker engines' it is an essential part of a Chess championship, as opposed to am engine-buidling contest.

[Milos]

White advantage increases as the average strength of the opponents increases. I am not saying that 50 Elo is a good assumption under the assumed conditions, for I have not had time to follow what the 3 of you are doing. But, you should not just assume that the advantage should be under 30 Elo.

True, but white advantage also decreases as the Elo gap between engines increases (i.e. higher white advantage artificially increases Elo gap), and in this hypothetical case we were discussing (of SF participating ICGA), gap would be quite substantial.

[michiguel]

I got this with the Ordo model

Code: Select all

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 50.0
rating[0]=0
rating[1]=-150
rating[2]=-200
rating[3]=-200
rating[4]=-250
rating[5]=-250
rating[6]=-300
rating[7]=-300
rating[8]=-400
rating[9]=-400
rating[10]=-400

won    = 827663
shared = 89338
loss   = 82999
total  = 1000000
won outright % = 82.8
won shared   % = 8.9

It is including white advantage and assuming 64% draw rate for equal strength.

Miguel

White advantage of 50Elo is unrealistic. In reality it is rarely above 30Elo.

White advantage increases as the average strength of the opponents increases. I am not saying that 50 Elo is a good assumption under the assumed conditions, for I have not had time to follow what the 3 of you are doing. But, you should not just assume that the advantage should be under 30 Elo.

I have the feeling that using books may increase white advantage. Anyway, simulations with 30 elo of white advantage are below.
Tie breaks are not applied. Those cases are the ones listed as "shared". When there was only one winner is listed as "outright".
I can try to implement some SB rules.

Miguel

Code: Select all

===========

Color when 1 plays white against 2

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200

won    = 622003
shared = 177919
loss   = 200078
total  = 1000000
won outright % = 62.2  <===============
won shared   % = 17.8

========

Reversed colors

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200

won    = 586411
shared = 186268
loss   = 227321
total  = 1000000
won outright % = 58.6 <===============
won shared   % = 18.6

[Milos]

I have the feeling that using books may increase white advantage. Anyway, simulations with 30 elo of white advantage are below.
Tie breaks are not applied. Those cases are the ones listed as "shared". When there was only one winner is listed as "outright".
I can try to implement some SB rules.

Miguel

Code: Select all

===========

Color when 1 plays white against 2

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200

won    = 622003
shared = 177919
loss   = 200078
total  = 1000000
won outright % = 62.2  <===============
won shared   % = 17.8

========

Reversed colors

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200

won    = 586411
shared = 186268
loss   = 227321
total  = 1000000
won outright % = 58.6 <===============
won shared   % = 18.6

Only if "white" engine had stronger book than the opponent, otherwise it's reduced. So overall I would say no effect.

In ICGA it's not SB but 2 games match with reversed colors and if still tied, a sudden death game with 20% more time for white.

[Harvey Williamson]

I have the feeling that using books may increase white advantage. Anyway, simulations with 30 elo of white advantage are below.
Tie breaks are not applied. Those cases are the ones listed as "shared". When there was only one winner is listed as "outright".
I can try to implement some SB rules.

Miguel

Code: Select all

===========

Color when 1 plays white against 2

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200

won    = 622003
shared = 177919
loss   = 200078
total  = 1000000
won outright % = 62.2  <===============
won shared   % = 17.8

========

Reversed colors

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200

won    = 586411
shared = 186268
loss   = 227321
total  = 1000000
won outright % = 58.6 <===============
won shared   % = 18.6

Only if "white" engine had stronger book than the opponent, otherwise it's reduced. So overall I would say no effect.

In ICGA it's not SB but 2 games match with reversed colors and if still tied, a sudden death game with 20% more time for white.

No it is not. It is either a Swiss or a double round robin. There is only a tie break IF 2 engines are tied at the end of the tournament.

[Laskos]

I have the feeling that using books may increase white advantage. Anyway, simulations with 30 elo of white advantage are below.
Tie breaks are not applied. Those cases are the ones listed as "shared". When there was only one winner is listed as "outright".
I can try to implement some SB rules.

Miguel

Code: Select all

===========

Color when 1 plays white against 2

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200

won    = 622003
shared = 177919
loss   = 200078
total  = 1000000
won outright % = 62.2  <===============
won shared   % = 17.8

========

Reversed colors

Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200

won    = 586411
shared = 186268
loss   = 227321
total  = 1000000
won outright % = 58.6 <===============
won shared   % = 18.6

I used drawelo of 200, which gives lower draw rate than your 64% for equal opponents. 64% seems a bit high, it's valid only for several top engines, the rest have lower draw rates. Can you use 56%? Probably my drawelo of 200 is a bit too low, though.