Knight equals 48 pawns?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
lkaufman
Posts: 4530
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Knight equals 48 pawns?

Post by lkaufman » Wed Feb 13, 2019 6:23 pm

I downloaded the latest Lc0 about a week ago on my laptop (with GPU 1060), and looked at its analysis (infinite mode) on lots of positions. Typically, in the opening evals are roughly double what they should be; in general a clean pawn up shows as about two pawns. However, the initial position with knight odds (b1 off) shows a fairly stable evaluation of around -48 pawns!!! Even if we divide by two (since the evals are double what they should be in normal positions), we get one knight = 24 pawns!! Moreover changing the knight to the g1 knight or to a bishop only made the score even more lopsided. I think that mathematically this is due to using the arctangent in the formula, but why would Lc0 choose to display such obvious nonsense, and is there any version that displays sensible evals? It shouldn't affect the play, just the display.
Komodo rules!

Werewolf
Posts: 1349
Joined: Thu Sep 18, 2008 8:24 pm

Re: Knight equals 48 pawns?

Post by Werewolf » Wed Feb 13, 2019 6:28 pm

It's definitely not showing probability odds, like Alpha Zero?

lkaufman
Posts: 4530
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Knight equals 48 pawns?

Post by lkaufman » Wed Feb 13, 2019 9:23 pm

Werewolf wrote:
Wed Feb 13, 2019 6:28 pm
It's definitely not showing probability odds, like Alpha Zero?
Nothing wrong with converting win probs to scores in pawn units, we do the same in Komodo MCTS. But we don't output nonsense scores like this. Looks like no one bothered to check whether the conversion formula made any sense.
Komodo rules!

User avatar
Laskos
Posts: 10949
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Knight equals 48 pawns?

Post by Laskos » Wed Feb 13, 2019 9:40 pm

lkaufman wrote:
Wed Feb 13, 2019 6:23 pm
I downloaded the latest Lc0 about a week ago on my laptop (with GPU 1060), and looked at its analysis (infinite mode) on lots of positions. Typically, in the opening evals are roughly double what they should be; in general a clean pawn up shows as about two pawns. However, the initial position with knight odds (b1 off) shows a fairly stable evaluation of around -48 pawns!!! Even if we divide by two (since the evals are double what they should be in normal positions), we get one knight = 24 pawns!! Moreover changing the knight to the g1 knight or to a bishop only made the score even more lopsided. I think that mathematically this is due to using the arctangent in the formula, but why would Lc0 choose to display such obvious nonsense, and is there any version that displays sensible evals? It shouldn't affect the play, just the display.
Sure arctan use and the calibration. What was with logistic a 4.00 eval meaning some very roughly exp(-eval) or here exp(-4) performance, now is 1/eval or 1/48 performance (very roughly). The tails of logistic are exponentially diminishing, of arctan inversely linearly (hyperbolic) diminishing. Maybe it's better to use with Leela the performance % display. I am not even sure when this arctan display became popular, earlier most engines showed some "real pawns", and the eval obeyed pretty well the logistic in this case (some engines like Texel even tuned the eval to logistic).

nabildanial
Posts: 126
Joined: Thu Jun 05, 2014 3:29 am
Location: Malaysia

Re: Knight equals 48 pawns?

Post by nabildanial » Thu Feb 14, 2019 12:57 am

Someone already opened an issue and proposed a better conversion formula on Lc0 Github a few days ago.

https://github.com/LeelaChessZero/lc0/issues/728

Werewolf
Posts: 1349
Joined: Thu Sep 18, 2008 8:24 pm

Re: Knight equals 48 pawns?

Post by Werewolf » Thu Feb 14, 2019 8:00 am

lkaufman wrote:
Wed Feb 13, 2019 9:23 pm
Werewolf wrote:
Wed Feb 13, 2019 6:28 pm
It's definitely not showing probability odds, like Alpha Zero?
Nothing wrong with converting win probs to scores in pawn units, we do the same in Komodo MCTS. But we don't output nonsense scores like this. Looks like no one bothered to check whether the conversion formula made any sense.
Test 50 looks like it might change this

https://medium.com/@veedrac/leela-chess ... 5896becfac

User avatar
George Tsavdaris
Posts: 1627
Joined: Thu Mar 09, 2006 11:35 am

Re: Knight equals 48 pawns?

Post by George Tsavdaris » Thu Feb 14, 2019 10:59 am

lkaufman wrote:
Wed Feb 13, 2019 6:23 pm
I downloaded the latest Lc0 about a week ago on my laptop (with GPU 1060), and looked at its analysis (infinite mode) on lots of positions. Typically, in the opening evals are roughly double what they should be; in general a clean pawn up shows as about two pawns. However, the initial position with knight odds (b1 off) shows a fairly stable evaluation of around -48 pawns!!! Even if we divide by two (since the evals are double what they should be in normal positions), we get one knight = 24 pawns!! Moreover changing the knight to the g1 knight or to a bishop only made the score even more lopsided. I think that mathematically this is due to using the arctangent in the formula, but why would Lc0 choose to display such obvious nonsense, and is there any version that displays sensible evals? It shouldn't affect the play, just the display.
Leela's evals should not be considered with normal thinking about pieces values and connections between Pawns-Knights etc but only as probabilities of winning/drawing.
Leela's evals are just a somewhat arbitrary function that maps her probabilities for draw/win to centiPawns and it's(assuming the weaker side has 0 probability to win):
http://pasted.co/1a396b09
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....

Branko Radovanovic
Posts: 75
Joined: Sat Sep 13, 2014 2:12 pm

Re: Knight equals 48 pawns?

Post by Branko Radovanovic » Thu Feb 14, 2019 5:01 pm

George Tsavdaris wrote:
Thu Feb 14, 2019 10:59 am
lkaufman wrote:
Wed Feb 13, 2019 6:23 pm
I downloaded the latest Lc0 about a week ago on my laptop (with GPU 1060), and looked at its analysis (infinite mode) on lots of positions. Typically, in the opening evals are roughly double what they should be; in general a clean pawn up shows as about two pawns. However, the initial position with knight odds (b1 off) shows a fairly stable evaluation of around -48 pawns!!! Even if we divide by two (since the evals are double what they should be in normal positions), we get one knight = 24 pawns!! Moreover changing the knight to the g1 knight or to a bishop only made the score even more lopsided. I think that mathematically this is due to using the arctangent in the formula, but why would Lc0 choose to display such obvious nonsense, and is there any version that displays sensible evals? It shouldn't affect the play, just the display.
Leela's evals should not be considered with normal thinking about pieces values and connections between Pawns-Knights etc but only as probabilities of winning/drawing.
Leela's evals are just a somewhat arbitrary function that maps her probabilities for draw/win to centiPawns and it's(assuming the weaker side has 0 probability to win):
http://pasted.co/1a396b09
Compare this to Houdini (http://www.cruxis.com/chess/manual/inde ... istory.htm):
The evaluations have again been calibrated to correlate directly with the win expectancy in the position. A +1.00 pawn advantage gives a 75% chance of winning the game against an equal opponent at blitz time control. At +1.50 the engine will win 90% of the time, and at +2.50 about 99% of the time. To win nearly 50% of the time, you need an advantage of about +0.60 pawn.
So, 75% chance of winning corresponds to +1 eval for Houdini and +6.7 eval for Leela, if I'm interpreting this correctly. That would indeed be highly distorted.

lkaufman
Posts: 4530
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Knight equals 48 pawns?

Post by lkaufman » Thu Feb 14, 2019 5:12 pm

It's even worse than this I think. The 75% figure for winning when up one pawn is not 75% score exp., but actual wins. If most of the other 25% are draws (as would surely be the case) the actual scoring percentage expectancy when a pawn up is about 85%, not 75%. The conversion formula should be adjusted accordingly. I think the figures are pretty similar for Komodo, although it varies a bit from version to version, and the time limit makes some difference too.
Komodo rules!

jhellis3
Posts: 488
Joined: Fri Aug 16, 2013 10:36 pm

Re: Knight equals 48 pawns?

Post by jhellis3 » Thu Feb 14, 2019 6:11 pm

So you disagree that the side starting the game a knight down is likely objectively lost?

Post Reply