Re: TCEC Division 1 results simulator

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
JJJ
Posts: 1277
Joined: Sat Apr 19, 2014 11:47 am

Re: TCEC Division 1 results simulator

Post by JJJ » Wed Aug 29, 2018 8:10 am

Laskos wrote:
Tue Aug 28, 2018 9:59 am
chrisw wrote:
Tue Aug 28, 2018 9:23 am
40 rounds so far, Division 1, sim predictions for being promoted to Premier Division:
Ethereal 90%
Chiron 83%
Fizbo 19%
Laser 3%

Code: Select all

Column 'First' is chance of winning, Column 'First two' is chance of being in first two
Engine,   Tournament Elo, Initial Elo, First, First two
Ethereal       3374         3334       0.562  0.909
Chiron         3351         3288       0.392  0.835
Fizbo          3301         3287       0.040  0.193
Laser          3247         3226       0.004  0.030
Fritz          3231         3226       0.001  0.014
ChessBrainVB   3247         3279       0.001  0.016
Jonny          3139         3140       0.000  0.000
Booot          3218         3272       0.000  0.002
Based on some not very well researched estimate elos, chances of winning the Premier Division would then be:

Code: Select all

Stockfish      3441         3441       0.572  
Komodo         3405         3405       0.186
Houdini        3400         3400       0.159  
Ethereal       3374         3374       0.055 
Chiron         3351         3351       0.019  
Fire           3326         3326       0.005 
Ginkgo         3322         3322       0.004  
Andscacs       3245         3245       0.000
About the second table. First, you seem to put Ethereal and Chiron too high. Second, generally, that in 56 games each, the winner is not one of the big three, seems to be a very slim possibility, much smaller than 8%. Third, Stockfish to win seems in above 70% to me. It has now "contempt" and is not underperforming in these round-robins with weaker engines. Gut feeling: SF say 72%, Kom 15%, Hou (if not updated) 11%, rest 2%.

And why not going further: SF above 90% to win the TCEC 13 (if Hou is not updated).
I would say the same thing. Houdini updated might have some good chance here.

The big 3 seems impossible to dethrone, even at this sample of game. And fire seems still above Chiron and Ethereal and might be update as well.

chrisw
Posts: 1732
Joined: Tue Apr 03, 2012 2:28 pm

Re: TCEC Division 1 results simulator

Post by chrisw » Wed Aug 29, 2018 10:44 am

cdani wrote:
Tue Aug 28, 2018 6:31 pm
Laskos wrote:
Tue Aug 28, 2018 9:59 am
chrisw wrote:
Tue Aug 28, 2018 9:23 am

Code: Select all

Andscacs       3245         3245       0.000
I think and I hope that Andscacs will not be last. 0.94 has no elo at CCRL for the moment. Also I have some improvements.
I also hope not!
Of course, the sim, with no cross table, is influenced only be the given elos, and will rank the engines in elo order.
If you have some rough estimate of Andsacs elo progress and a possible elo estimate, we can use that figure instead

chrisw
Posts: 1732
Joined: Tue Apr 03, 2012 2:28 pm

Re: TCEC Division 1 results simulator

Post by chrisw » Wed Aug 29, 2018 11:01 am

Laskos wrote:
Tue Aug 28, 2018 9:59 am
chrisw wrote:
Tue Aug 28, 2018 9:23 am
40 rounds so far, Division 1, sim predictions for being promoted to Premier Division:
Ethereal 90%
Chiron 83%
Fizbo 19%
Laser 3%

Code: Select all

Column 'First' is chance of winning, Column 'First two' is chance of being in first two
Engine,   Tournament Elo, Initial Elo, First, First two
Ethereal       3374         3334       0.562  0.909
Chiron         3351         3288       0.392  0.835
Fizbo          3301         3287       0.040  0.193
Laser          3247         3226       0.004  0.030
Fritz          3231         3226       0.001  0.014
ChessBrainVB   3247         3279       0.001  0.016
Jonny          3139         3140       0.000  0.000
Booot          3218         3272       0.000  0.002
Based on some not very well researched estimate elos, chances of winning the Premier Division would then be:

Code: Select all

Stockfish      3441         3441       0.572  
Komodo         3405         3405       0.186
Houdini        3400         3400       0.159  
Ethereal       3374         3374       0.055 
Chiron         3351         3351       0.019  
Fire           3326         3326       0.005 
Ginkgo         3322         3322       0.004  
Andscacs       3245         3245       0.000
About the second table. First, you seem to put Ethereal and Chiron too high. Second, generally, that in 56 games each, the winner is not one of the big three, seems to be a very slim possibility, much smaller than 8%. Third, Stockfish to win seems in above 70% to me. It has now "contempt" and is not underperforming in these round-robins with weaker engines. Gut feeling: SF say 72%, Kom 15%, Hou (if not updated) 11%, rest 2%.

And why not going further: SF above 90% to win the TCEC 13 (if Hou is not updated).
Ethereal and Chiron elos are being initialised to their given elo from the previous round plus 50% of the tournament elo gain, where:
tournament game = sum of [+/-400] / game count including draws
elo = initial elo + 0.5 * tournament gain

The elos of the other six engines are more likely to be problematic (too low, because progress).
Progress is tricky to guess at. At the moment I'ld go for:
Stockfish (++) because worked on continuously and reporting progress
Komodo (+) because worked on, but the side project MCTS may be distracting
Houdini (++) or (+) or (??) because I don't know
Fire (??) no idea at the moment
Ginggo (??)
Andsacs (+) or (++) because worked on

The Monte Carlo rollouts for the Premier are adjusted for the doubled game count (I just checked that I remembered to do that), and that will have effect of increasing the top winning chances for the sim.

chrisw
Posts: 1732
Joined: Tue Apr 03, 2012 2:28 pm

Re: TCEC Division 1 results simulator

Post by chrisw » Wed Aug 29, 2018 11:34 am

53 rounds so far, Division 1, sim prediction of promotion:
Ethereal 95%
Chiron 94%
Fizbo 7%

First tie-break (assuming zero disconnects) is on direct result between the two, where Chiron is currently ahead against Ethereal.

Code: Select all

Column 'First' is chance of winning, Column 'First two' is chance of being in first two
Engine,   Tournament Elo, Initial Elo, First, First two
Ethereal       3378         3334       0.522  0.958
Chiron         3365         3288       0.467  0.946
Fizbo          3289         3287       0.009  0.072
ChessBrainVB   3256         3279       0.001  0.017
Fritz          3228         3226       0.000  0.006
Jonny          3152         3140       0.000  0.000
Booot          3216         3272       0.000  0.000
Laser          3210         3226       0.000  0.002
TCEC cross table

Code: Select all

 1 Chiron S13        3340 9.0 13 56.25 0 +49 69.2 ·· 1= == =1 1= 0  =1 11
 2 Ethereal 10.85    3341 9.0 13 52.75 0 +44 69.2 0= ·· == 11 1  =1 == 11
 3 Fizbo 2           3284 7.5 14 48.25 0 +19 53.6 == == ·· == == 1= == ==
 4 ChessBrainVB 3.70 3300 6.0 13 37.50 0 -24 46.2 =0 00 == ·· 1= =1 1= 0 
 5 Fritz 16.10       3294 6.0 13 35.50 0 -23 46.2 0= 0  == 0= ·· == 1= 1=
 6 Laser 180818      3194 5.5 13 36.25 0 +28 42.3 1  =0 0= =0 == ·· == ==
 7 Booot 6.3.1       3273 5.5 14 37.00 2 -37 39.3 =0 == == 0= 0= == ·· ==
 8 Jonny 8.1         3274 4.5 13 27.50 0 -55 34.6 00 00 == 1  0= == == ··

Milos
Posts: 3372
Joined: Wed Nov 25, 2009 12:47 am

Re: TCEC Division 1 results simulator

Post by Milos » Thu Aug 30, 2018 12:18 am

chrisw wrote:
Wed Aug 29, 2018 11:01 am
Laskos wrote:
Tue Aug 28, 2018 9:59 am
chrisw wrote:
Tue Aug 28, 2018 9:23 am
40 rounds so far, Division 1, sim predictions for being promoted to Premier Division:
Ethereal 90%
Chiron 83%
Fizbo 19%
Laser 3%

Code: Select all

Column 'First' is chance of winning, Column 'First two' is chance of being in first two
Engine,   Tournament Elo, Initial Elo, First, First two
Ethereal       3374         3334       0.562  0.909
Chiron         3351         3288       0.392  0.835
Fizbo          3301         3287       0.040  0.193
Laser          3247         3226       0.004  0.030
Fritz          3231         3226       0.001  0.014
ChessBrainVB   3247         3279       0.001  0.016
Jonny          3139         3140       0.000  0.000
Booot          3218         3272       0.000  0.002
Based on some not very well researched estimate elos, chances of winning the Premier Division would then be:

Code: Select all

Stockfish      3441         3441       0.572  
Komodo         3405         3405       0.186
Houdini        3400         3400       0.159  
Ethereal       3374         3374       0.055 
Chiron         3351         3351       0.019  
Fire           3326         3326       0.005 
Ginkgo         3322         3322       0.004  
Andscacs       3245         3245       0.000
About the second table. First, you seem to put Ethereal and Chiron too high. Second, generally, that in 56 games each, the winner is not one of the big three, seems to be a very slim possibility, much smaller than 8%. Third, Stockfish to win seems in above 70% to me. It has now "contempt" and is not underperforming in these round-robins with weaker engines. Gut feeling: SF say 72%, Kom 15%, Hou (if not updated) 11%, rest 2%.

And why not going further: SF above 90% to win the TCEC 13 (if Hou is not updated).
Ethereal and Chiron elos are being initialised to their given elo from the previous round plus 50% of the tournament elo gain, where:
tournament game = sum of [+/-400] / game count including draws
elo = initial elo + 0.5 * tournament gain

The elos of the other six engines are more likely to be problematic (too low, because progress).
Progress is tricky to guess at. At the moment I'ld go for:
Stockfish (++) because worked on continuously and reporting progress
Komodo (+) because worked on, but the side project MCTS may be distracting
Houdini (++) or (+) or (??) because I don't know
Fire (??) no idea at the moment
Ginggo (??)
Andsacs (+) or (++) because worked on

The Monte Carlo rollouts for the Premier are adjusted for the doubled game count (I just checked that I remembered to do that), and that will have effect of increasing the top winning chances for the sim.
Just a small illustration how ridiculous are your assumptions (regarding Premier Division):
http://talkchess.com/forum3/viewtopic.p ... 31#p772531

jdart
Posts: 3786
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: TCEC Division 1 results simulator

Post by jdart » Thu Aug 30, 2018 3:27 am

Is TCEC down? I am trying to load http://tcec.chessdom.com/season13/live.php and getting no response.

--Jon

AndrewGrant
Posts: 467
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: TCEC Division 1 results simulator

Post by AndrewGrant » Thu Aug 30, 2018 4:02 am

jdart wrote:
Thu Aug 30, 2018 3:27 am
Is TCEC down? I am trying to load http://tcec.chessdom.com/season13/live.php and getting no response.

--Jon
http://tcec.chessdom.com/beta/live.html

Admins are moving around the GUIs

Uri Blass
Posts: 8506
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: TCEC Division 1 results simulator

Post by Uri Blass » Thu Aug 30, 2018 6:53 am

Milos wrote:
Thu Aug 30, 2018 12:18 am
chrisw wrote:
Wed Aug 29, 2018 11:01 am
Laskos wrote:
Tue Aug 28, 2018 9:59 am
chrisw wrote:
Tue Aug 28, 2018 9:23 am
40 rounds so far, Division 1, sim predictions for being promoted to Premier Division:
Ethereal 90%
Chiron 83%
Fizbo 19%
Laser 3%

Code: Select all

Column 'First' is chance of winning, Column 'First two' is chance of being in first two
Engine,   Tournament Elo, Initial Elo, First, First two
Ethereal       3374         3334       0.562  0.909
Chiron         3351         3288       0.392  0.835
Fizbo          3301         3287       0.040  0.193
Laser          3247         3226       0.004  0.030
Fritz          3231         3226       0.001  0.014
ChessBrainVB   3247         3279       0.001  0.016
Jonny          3139         3140       0.000  0.000
Booot          3218         3272       0.000  0.002
Based on some not very well researched estimate elos, chances of winning the Premier Division would then be:

Code: Select all

Stockfish      3441         3441       0.572  
Komodo         3405         3405       0.186
Houdini        3400         3400       0.159  
Ethereal       3374         3374       0.055 
Chiron         3351         3351       0.019  
Fire           3326         3326       0.005 
Ginkgo         3322         3322       0.004  
Andscacs       3245         3245       0.000
About the second table. First, you seem to put Ethereal and Chiron too high. Second, generally, that in 56 games each, the winner is not one of the big three, seems to be a very slim possibility, much smaller than 8%. Third, Stockfish to win seems in above 70% to me. It has now "contempt" and is not underperforming in these round-robins with weaker engines. Gut feeling: SF say 72%, Kom 15%, Hou (if not updated) 11%, rest 2%.

And why not going further: SF above 90% to win the TCEC 13 (if Hou is not updated).
Ethereal and Chiron elos are being initialised to their given elo from the previous round plus 50% of the tournament elo gain, where:
tournament game = sum of [+/-400] / game count including draws
elo = initial elo + 0.5 * tournament gain

The elos of the other six engines are more likely to be problematic (too low, because progress).
Progress is tricky to guess at. At the moment I'ld go for:
Stockfish (++) because worked on continuously and reporting progress
Komodo (+) because worked on, but the side project MCTS may be distracting
Houdini (++) or (+) or (??) because I don't know
Fire (??) no idea at the moment
Ginggo (??)
Andsacs (+) or (++) because worked on

The Monte Carlo rollouts for the Premier are adjusted for the doubled game count (I just checked that I remembered to do that), and that will have effect of increasing the top winning chances for the sim.
Just a small illustration how ridiculous are your assumptions (regarding Premier Division):
http://talkchess.com/forum3/viewtopic.p ... 31#p772531
Note that TCEC is a different type of competition than 3+1 blitz.
I do not claim that Ethereal is better than Fire at TCEC conditions but only that I do not know.
Here is CCRL 40/40
http://www.computerchess.org.uk/ccrl/40 ... t_all.html

I see
Fire 7.1 64-bit 4CPU 3326
Deep Shredder 13 64-bit 4CPU 3287
Ethereal 10.55 64-bit 4CPU 3283

I guess new ethereal is better than 10.55 and shredder13 so the gap at 40/40 seem to be smaller than 40 elo.

Milos
Posts: 3372
Joined: Wed Nov 25, 2009 12:47 am

Re: TCEC Division 1 results simulator

Post by Milos » Thu Aug 30, 2018 11:41 am

Uri Blass wrote:
Thu Aug 30, 2018 6:53 am
Milos wrote:
Thu Aug 30, 2018 12:18 am
chrisw wrote:
Wed Aug 29, 2018 11:01 am
Laskos wrote:
Tue Aug 28, 2018 9:59 am
chrisw wrote:
Tue Aug 28, 2018 9:23 am
40 rounds so far, Division 1, sim predictions for being promoted to Premier Division:
Ethereal 90%
Chiron 83%
Fizbo 19%
Laser 3%

Code: Select all

Column 'First' is chance of winning, Column 'First two' is chance of being in first two
Engine,   Tournament Elo, Initial Elo, First, First two
Ethereal       3374         3334       0.562  0.909
Chiron         3351         3288       0.392  0.835
Fizbo          3301         3287       0.040  0.193
Laser          3247         3226       0.004  0.030
Fritz          3231         3226       0.001  0.014
ChessBrainVB   3247         3279       0.001  0.016
Jonny          3139         3140       0.000  0.000
Booot          3218         3272       0.000  0.002
Based on some not very well researched estimate elos, chances of winning the Premier Division would then be:

Code: Select all

Stockfish      3441         3441       0.572  
Komodo         3405         3405       0.186
Houdini        3400         3400       0.159  
Ethereal       3374         3374       0.055 
Chiron         3351         3351       0.019  
Fire           3326         3326       0.005 
Ginkgo         3322         3322       0.004  
Andscacs       3245         3245       0.000
About the second table. First, you seem to put Ethereal and Chiron too high. Second, generally, that in 56 games each, the winner is not one of the big three, seems to be a very slim possibility, much smaller than 8%. Third, Stockfish to win seems in above 70% to me. It has now "contempt" and is not underperforming in these round-robins with weaker engines. Gut feeling: SF say 72%, Kom 15%, Hou (if not updated) 11%, rest 2%.

And why not going further: SF above 90% to win the TCEC 13 (if Hou is not updated).
Ethereal and Chiron elos are being initialised to their given elo from the previous round plus 50% of the tournament elo gain, where:
tournament game = sum of [+/-400] / game count including draws
elo = initial elo + 0.5 * tournament gain

The elos of the other six engines are more likely to be problematic (too low, because progress).
Progress is tricky to guess at. At the moment I'ld go for:
Stockfish (++) because worked on continuously and reporting progress
Komodo (+) because worked on, but the side project MCTS may be distracting
Houdini (++) or (+) or (??) because I don't know
Fire (??) no idea at the moment
Ginggo (??)
Andsacs (+) or (++) because worked on

The Monte Carlo rollouts for the Premier are adjusted for the doubled game count (I just checked that I remembered to do that), and that will have effect of increasing the top winning chances for the sim.
Just a small illustration how ridiculous are your assumptions (regarding Premier Division):
http://talkchess.com/forum3/viewtopic.p ... 31#p772531
Note that TCEC is a different type of competition than 3+1 blitz.
I do not claim that Ethereal is better than Fire at TCEC conditions but only that I do not know.
Here is CCRL 40/40
http://www.computerchess.org.uk/ccrl/40 ... t_all.html

I see
Fire 7.1 64-bit 4CPU 3326
Deep Shredder 13 64-bit 4CPU 3287
Ethereal 10.55 64-bit 4CPU 3283

I guess new ethereal is better than 10.55 and shredder13 so the gap at 40/40 seem to be smaller than 40 elo.
CCRL 40/40 is irrelevant. Ethereal only played 400 games there.
It played 50 games in TCEC and ppl nake even more ridiculous conclusions. It'a hype and nothing else. Chance for Ethereal to finish 4th in primary division is less than 30%. To finish in top 3 is less than 1%.

Milos
Posts: 3372
Joined: Wed Nov 25, 2009 12:47 am

Re: TCEC Division 1 results simulator

Post by Milos » Thu Aug 30, 2018 11:43 am

:twisted: op
Uri Blass wrote:
Thu Aug 30, 2018 6:53 am
Milos wrote:
Thu Aug 30, 2018 12:18 am
chrisw wrote:
Wed Aug 29, 2018 11:01 am
Laskos wrote:
Tue Aug 28, 2018 9:59 am
chrisw wrote:
Tue Aug 28, 2018 9:23 am
40 rounds so far, Division 1, sim predictions for being promoted to Premier Division:
Ethereal 90%
Chiron 83%
Fizbo 19%
Laser 3%

Code: Select all

Column 'First' is chance of winning, Column 'First two' is chance of being in first two
Engine,   Tournament Elo, Initial Elo, First, First two
Ethereal       3374         3334       0.562  0.909
Chiron         3351         3288       0.392  0.835
Fizbo          3301         3287       0.040  0.193
Laser          3247         3226       0.004  0.030
Fritz          3231         3226       0.001  0.014
ChessBrainVB   3247         3279       0.001  0.016
Jonny          3139         3140       0.000  0.000
Booot          3218         3272       0.000  0.002
Based on some not very well researched estimate elos, chances of winning the Premier Division would then be:

Code: Select all

Stockfish      3441         3441       0.572  
Komodo         3405         3405       0.186
Houdini        3400         3400       0.159  
Ethereal       3374         3374       0.055 
Chiron         3351         3351       0.019  
Fire           3326         3326       0.005 
Ginkgo         3322         3322       0.004  
Andscacs       3245         3245       0.000
About the second table. First, you seem to put Ethereal and Chiron too high. Second, generally, that in 56 games each, the winner is not one of the big three, seems to be a very slim possibility, much smaller than 8%. Third, Stockfish to win seems in above 70% to me. It has now "contempt" and is not underperforming in these round-robins with weaker engines. Gut feeling: SF say 72%, Kom 15%, Hou (if not updated) 11%, rest 2%.

And why not going further: SF above 90% to win the TCEC 13 (if Hou is not updated).
Ethereal and Chiron elos are being initialised to their given elo from the previous round plus 50% of the tournament elo gain, where:
tournament game = sum of [+/-400] / game count including draws
elo = initial elo + 0.5 * tournament gain

The elos of the other six engines are more likely to be problematic (too low, because progress).
Progress is tricky to guess at. At the moment I'ld go for:
Stockfish (++) because worked on continuously and reporting progress
Komodo (+) because worked on, but the side project MCTS may be distracting
Houdini (++) or (+) or (??) because I don't know
Fire (??) no idea at the moment
Ginggo (??)
Andsacs (+) or (++) because worked on

The Monte Carlo rollouts for the Premier are adjusted for the doubled game count (I just checked that I remembered to do that), and that will have effect of increasing the top winning chances for the sim.
Just a small illustration how ridiculous are your assumptions (regarding Premier Division):
http://talkchess.com/forum3/viewtopic.p ... 31#p772531
Note that TCEC is a different type of competition than 3+1 blitz.
I do not claim that Ethereal is better than Fire at TCEC conditions but only that I do not know.
Here is CCRL 40/40
http://www.computerchess.org.uk/ccrl/40 ... t_all.html

I see
Fire 7.1 64-bit 4CPU 3326
Deep Shredder 13 64-bit 4CPU 3287
Ethereal 10.55 64-bit 4CPU 3283

I guess new ethereal is better than 10.55 and shredder13 so the gap at 40/40 seem to be smaller than 40 elo.
Oh give me a break and try to say something useful for a change. Both Fire and Ethereal have identical LazySMP as SF, and scale pretty much identically. These claims that we don't know how something behaves at 10 or 40x longer TC is just BS. We do know. There is just compression of rating differences. There is never an inversion. No one ever showed one with any reasonably strong A/B engine.
CCRL 40/40 is irrelevant. Ethereal only played 400 games there. Probably real difference vs Fire is 50-60 Elo instead of 70-80. And that's perfectly fine for 10x longer TC. And it's a simple fact that you compress rating differences with longer TC but you also compress error margins. So probability of marginal events happening remains the same.
It played 50 games in TCEC and ppl make even more ridiculous conclusions. It's a hype and nothing else. Chance for Ethereal to finish 4th in primary division is less than 20%. To finish in top 3 is less than 2%.

Post Reply