Knight equals 48 pawns?

[lkaufman]

I downloaded the latest Lc0 about a week ago on my laptop (with GPU 1060), and looked at its analysis (infinite mode) on lots of positions. Typically, in the opening evals are roughly double what they should be; in general a clean pawn up shows as about two pawns. However, the initial position with knight odds (b1 off) shows a fairly stable evaluation of around -48 pawns!!! Even if we divide by two (since the evals are double what they should be in normal positions), we get one knight = 24 pawns!! Moreover changing the knight to the g1 knight or to a bishop only made the score even more lopsided. I think that mathematically this is due to using the arctangent in the formula, but why would Lc0 choose to display such obvious nonsense, and is there any version that displays sensible evals? It shouldn't affect the play, just the display.

[Werewolf]

It's definitely not showing probability odds, like Alpha Zero?

[lkaufman]

It's definitely not showing probability odds, like Alpha Zero?

Nothing wrong with converting win probs to scores in pawn units, we do the same in Komodo MCTS. But we don't output nonsense scores like this. Looks like no one bothered to check whether the conversion formula made any sense.

[Laskos]

I downloaded the latest Lc0 about a week ago on my laptop (with GPU 1060), and looked at its analysis (infinite mode) on lots of positions. Typically, in the opening evals are roughly double what they should be; in general a clean pawn up shows as about two pawns. However, the initial position with knight odds (b1 off) shows a fairly stable evaluation of around -48 pawns!!! Even if we divide by two (since the evals are double what they should be in normal positions), we get one knight = 24 pawns!! Moreover changing the knight to the g1 knight or to a bishop only made the score even more lopsided. I think that mathematically this is due to using the arctangent in the formula, but why would Lc0 choose to display such obvious nonsense, and is there any version that displays sensible evals? It shouldn't affect the play, just the display.

Sure arctan use and the calibration. What was with logistic a 4.00 eval meaning some very roughly exp(-eval) or here exp(-4) performance, now is 1/eval or 1/48 performance (very roughly). The tails of logistic are exponentially diminishing, of arctan inversely linearly (hyperbolic) diminishing. Maybe it's better to use with Leela the performance % display. I am not even sure when this arctan display became popular, earlier most engines showed some "real pawns", and the eval obeyed pretty well the logistic in this case (some engines like Texel even tuned the eval to logistic).

[nabildanial]

Someone already opened an issue and proposed a better conversion formula on Lc0 Github a few days ago.

https://github.com/LeelaChessZero/lc0/issues/728

[Werewolf]

It's definitely not showing probability odds, like Alpha Zero?

Nothing wrong with converting win probs to scores in pawn units, we do the same in Komodo MCTS. But we don't output nonsense scores like this. Looks like no one bothered to check whether the conversion formula made any sense.

Test 50 looks like it might change this

https://medium.com/@veedrac/leela-chess ... 5896becfac

[George Tsavdaris]

I downloaded the latest Lc0 about a week ago on my laptop (with GPU 1060), and looked at its analysis (infinite mode) on lots of positions. Typically, in the opening evals are roughly double what they should be; in general a clean pawn up shows as about two pawns. However, the initial position with knight odds (b1 off) shows a fairly stable evaluation of around -48 pawns!!! Even if we divide by two (since the evals are double what they should be in normal positions), we get one knight = 24 pawns!! Moreover changing the knight to the g1 knight or to a bishop only made the score even more lopsided. I think that mathematically this is due to using the arctangent in the formula, but why would Lc0 choose to display such obvious nonsense, and is there any version that displays sensible evals? It shouldn't affect the play, just the display.

Leela's evals should not be considered with normal thinking about pieces values and connections between Pawns-Knights etc but only as probabilities of winning/drawing.
Leela's evals are just a somewhat arbitrary function that maps her probabilities for draw/win to centiPawns and it's(assuming the weaker side has 0 probability to win):
http://pasted.co/1a396b09

[Branko Radovanovic]

I downloaded the latest Lc0 about a week ago on my laptop (with GPU 1060), and looked at its analysis (infinite mode) on lots of positions. Typically, in the opening evals are roughly double what they should be; in general a clean pawn up shows as about two pawns. However, the initial position with knight odds (b1 off) shows a fairly stable evaluation of around -48 pawns!!! Even if we divide by two (since the evals are double what they should be in normal positions), we get one knight = 24 pawns!! Moreover changing the knight to the g1 knight or to a bishop only made the score even more lopsided. I think that mathematically this is due to using the arctangent in the formula, but why would Lc0 choose to display such obvious nonsense, and is there any version that displays sensible evals? It shouldn't affect the play, just the display.

Leela's evals should not be considered with normal thinking about pieces values and connections between Pawns-Knights etc but only as probabilities of winning/drawing.
Leela's evals are just a somewhat arbitrary function that maps her probabilities for draw/win to centiPawns and it's(assuming the weaker side has 0 probability to win):
http://pasted.co/1a396b09

Compare this to Houdini (http://www.cruxis.com/chess/manual/inde ... istory.htm):

The evaluations have again been calibrated to correlate directly with the win expectancy in the position. A +1.00 pawn advantage gives a 75% chance of winning the game against an equal opponent at blitz time control. At +1.50 the engine will win 90% of the time, and at +2.50 about 99% of the time. To win nearly 50% of the time, you need an advantage of about +0.60 pawn.

So, 75% chance of winning corresponds to +1 eval for Houdini and +6.7 eval for Leela, if I'm interpreting this correctly. That would indeed be highly distorted.

[lkaufman]

It's even worse than this I think. The 75% figure for winning when up one pawn is not 75% score exp., but actual wins. If most of the other 25% are draws (as would surely be the case) the actual scoring percentage expectancy when a pawn up is about 85%, not 75%. The conversion formula should be adjusted accordingly. I think the figures are pretty similar for Komodo, although it varies a bit from version to version, and the time limit makes some difference too.

[jhellis3]

So you disagree that the side starting the game a knight down is likely objectively lost?