Are draws hard to predict?

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
jp
Posts: 259
Joined: Mon Apr 23, 2018 5:54 am

Re: Are draws hard to predict?

Post by jp » Fri Nov 30, 2018 3:53 pm

chrisw wrote:
Fri Nov 30, 2018 1:40 pm
jp wrote:
Fri Nov 30, 2018 12:03 pm
What do Leela etc. do re. W/D/L vs. W/L?
treat as one continuous value, I think, last time I looked. value is not treated as categorical, the trained network output is a value between 0 and 1, representing win probablity.
I've been wanting to check what that means. Does 0.7 mean win prob. 70% and combined draw & loss prob. 30% (not distinguishing between D & L), or does it mean something else (e.g. expected score, Pr(win)+ (1/2).Pr(draw))?

chrisw
Posts: 782
Joined: Tue Apr 03, 2012 2:28 pm

Re: Are draws hard to predict?

Post by chrisw » Fri Nov 30, 2018 5:20 pm

jp wrote:
Fri Nov 30, 2018 3:53 pm
chrisw wrote:
Fri Nov 30, 2018 1:40 pm
jp wrote:
Fri Nov 30, 2018 12:03 pm
What do Leela etc. do re. W/D/L vs. W/L?
treat as one continuous value, I think, last time I looked. value is not treated as categorical, the trained network output is a value between 0 and 1, representing win probablity.
I've been wanting to check what that means. Does 0.7 mean win prob. 70% and combined draw & loss prob. 30% (not distinguishing between D & L), or does it mean something else (e.g. expected score, Pr(win)+ (1/2).Pr(draw))?
it means the same as a typical AB program score, they are convertible between centipawns and win chances via formula. okay, so you ask what is “win chance”, good question, nobody programming cares much, the idea is just to maximise the magic number.

for AB the number comes from adding up some chessic features. The fact, for example, that you will win 100% of your games against any entity, however strong, if you are +5 for KRK or +9 for KQK, tells you that this AB score is not really a “win chance”.
Likewise NN, might say 90% or something, when you know its 100% against any entity.

The NN win prob, is actually the result of hitting the network during training with either a 1 or a 0 or a -1, in similar positions, say 10,000 times, when it got hit maybe 6000 times with 1, 3000 times with 0 and 1000 times with -1. Okay I cheat a little, but this going to give 0.9 or whatever. And this means? errrrm, doesn’t matter, what counts is that this 0.9 is greater than 0.87 someplace else, so the algorithm “knows what to do”.

Ultimately, the problem rests with those endusers, encouraged by those who really ought to know better, who spread the narrative that these scores are exact and accurate and correct and meaningful and all you need is mega silicon bucks plus Stockfish or whatever. Then you have god and can prove anything.

So to answer your question. The value is a magic number whose task is to provide direction by comparing it with other magic numbers. Avoid trying to treat it as anything absolute, cos it isn’t.
Chris Whittington
Ontological Terrorist
Lead Philosopher Computer Chess
Comrade Commissar Aspergers Liberation Front
Ostranenye Catalyst
Zero tolerance personal comments
5-4-8 tritype, INTP, BCF 221
facebook: chris.whittington.522
twitter: Skool_Dog

jp
Posts: 259
Joined: Mon Apr 23, 2018 5:54 am

Re: Are draws hard to predict?

Post by jp » Sat Dec 01, 2018 5:03 pm

chrisw wrote:
Fri Nov 30, 2018 5:20 pm
it means the same as a typical AB program score, they are convertible between centipawns and win chances via formula. okay, so you ask what is “win chance”, good question, nobody programming cares much, the idea is just to maximise the magic number.

for AB the number comes from adding up some chessic features.

Ultimately, the problem rests with those endusers, encouraged by those who really ought to know better, who spread the narrative that these scores are exact and accurate and correct and meaningful and all you need is mega silicon bucks plus Stockfish or whatever. Then you have god and can prove anything.

So to answer your question. The value is a magic number whose task is to provide direction by comparing it with other magic numbers. Avoid trying to treat it as anything absolute, cos it isn’t.
Yes. On converting between cP & win chance: this is a question re. Lc0 evaluation. In cP, Lc0 always overestimates. Some people claim this is just because of the conversion formula used. It's another thing I wanted to look into. I don't know what formula is used or if it's "standard". Those people would then claim Lc0's eval. is really okay, even if it looks crazy.

chrisw
Posts: 782
Joined: Tue Apr 03, 2012 2:28 pm

Re: Are draws hard to predict?

Post by chrisw » Sat Dec 01, 2018 8:42 pm

jp wrote:
Sat Dec 01, 2018 5:03 pm
chrisw wrote:
Fri Nov 30, 2018 5:20 pm
it means the same as a typical AB program score, they are convertible between centipawns and win chances via formula. okay, so you ask what is “win chance”, good question, nobody programming cares much, the idea is just to maximise the magic number.

for AB the number comes from adding up some chessic features.

Ultimately, the problem rests with those endusers, encouraged by those who really ought to know better, who spread the narrative that these scores are exact and accurate and correct and meaningful and all you need is mega silicon bucks plus Stockfish or whatever. Then you have god and can prove anything.

So to answer your question. The value is a magic number whose task is to provide direction by comparing it with other magic numbers. Avoid trying to treat it as anything absolute, cos it isn’t.
Yes. On converting between cP & win chance: this is a question re. Lc0 evaluation. In cP, Lc0 always overestimates. Some people claim this is just because of the conversion formula used. It's another thing I wanted to look into. I don't know what formula is used or if it's "standard". Those people would then claim Lc0's eval. is really okay, even if it looks crazy.
the conversion is empirical, I don’t remember who first did the statistic equivalences, it may have been Vas. It’s unlikely any rigorous connection between the two has ever been established. LC0 evaluation is optimistic usually, because, er, dunno. Not really thought about it.
Chris Whittington
Ontological Terrorist
Lead Philosopher Computer Chess
Comrade Commissar Aspergers Liberation Front
Ostranenye Catalyst
Zero tolerance personal comments
5-4-8 tritype, INTP, BCF 221
facebook: chris.whittington.522
twitter: Skool_Dog

User avatar
cdani
Posts: 2079
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: Are draws hard to predict?

Post by cdani » Sun Dec 02, 2018 3:31 pm

chrisw wrote:
Sat Dec 01, 2018 8:42 pm
...LC0 evaluation is optimistic usually, because, er, dunno. Not really thought about it.
I guess it's the result of that the positions it reaches tend to work against himself in selfplay.

jp
Posts: 259
Joined: Mon Apr 23, 2018 5:54 am

Re: Are draws hard to predict?

Post by jp » Mon Dec 03, 2018 11:45 pm

cdani wrote:
Sun Dec 02, 2018 3:31 pm
chrisw wrote:
Sat Dec 01, 2018 8:42 pm
...LC0 evaluation is optimistic usually, because, er, dunno. Not really thought about it.
I guess it's the result of that the positions it reaches tend to work against himself in selfplay.
Another thing is Kai found the Lc0 eval is asymmetric:
http://talkchess.com/forum3/viewtopic.php?f=2&t=68072

Now looking at that thread again I see Alexander replied.
crem wrote:
Wed Jul 25, 2018 4:00 pm
Interesting! Our Q to centipawns formula is symmetric (namely, 290.680623072 * tan(1.548090806 * Q)), where Q is between -1 and 1).
The reason for that bias is probably Lc0 being overly optimistic, due to blunders, not knowing how to win won endgames etc.

chrisw
Posts: 782
Joined: Tue Apr 03, 2012 2:28 pm

Re: Are draws hard to predict?

Post by chrisw » Tue Dec 04, 2018 7:32 pm

jp wrote:
Mon Dec 03, 2018 11:45 pm
cdani wrote:
Sun Dec 02, 2018 3:31 pm
chrisw wrote:
Sat Dec 01, 2018 8:42 pm
...LC0 evaluation is optimistic usually, because, er, dunno. Not really thought about it.
I guess it's the result of that the positions it reaches tend to work against himself in selfplay.
Another thing is Kai found the Lc0 eval is asymmetric:
http://talkchess.com/forum3/viewtopic.php?f=2&t=68072

Now looking at that thread again I see Alexander replied.
crem wrote:
Wed Jul 25, 2018 4:00 pm
Interesting! Our Q to centipawns formula is symmetric (namely, 290.680623072 * tan(1.548090806 * Q)), where Q is between -1 and 1).
The reason for that bias is probably Lc0 being overly optimistic, due to blunders, not knowing how to win won endgames etc.
It would be a thing if the words asymmetric, optimistic and unbalanced were defined and consistently used. Optimistic tends to be used okay, but the others not.
Chris Whittington
Ontological Terrorist
Lead Philosopher Computer Chess
Comrade Commissar Aspergers Liberation Front
Ostranenye Catalyst
Zero tolerance personal comments
5-4-8 tritype, INTP, BCF 221
facebook: chris.whittington.522
twitter: Skool_Dog

jp
Posts: 259
Joined: Mon Apr 23, 2018 5:54 am

Re: Are draws hard to predict?

Post by jp » Tue Dec 04, 2018 9:50 pm

True. But his graphs at least show what he means by "asymmetric".

chrisw
Posts: 782
Joined: Tue Apr 03, 2012 2:28 pm

Re: Are draws hard to predict?

Post by chrisw » Tue Dec 04, 2018 10:38 pm

jp wrote:
Tue Dec 04, 2018 9:50 pm
True. But his graphs at least show what he means by "asymmetric".
Maybe, but it’s not the “traditional” use of the term in comp chess. Where it means the evaluation and/or search are skewed depending on whether the engine is on move or not. eg, the evaulation and/or search are different depending on odd or even ply. I didn’t look for a while, but I don’t think LC0 has any odd/even ply differences.

Laskos graphs show either optimism or else an inability to convert a higher earlier score into a later win. Or both.
Chris Whittington
Ontological Terrorist
Lead Philosopher Computer Chess
Comrade Commissar Aspergers Liberation Front
Ostranenye Catalyst
Zero tolerance personal comments
5-4-8 tritype, INTP, BCF 221
facebook: chris.whittington.522
twitter: Skool_Dog

Post Reply