Knight equals 48 pawns?

[Laskos]

So you disagree that the side starting the game a knight down is likely objectively lost?

Against me?

[jhellis3]

The job of a chess engine eval is not to provide a value based upon inferior play of opponent, but an objective measure of the position (assuming best play as far as it knows), at least in my eyes. Now if someone wants to change that definition, they are certainly free to do so, but I doubt many will find their idiosyncrasies particularly valuable. However, I suspect you already know all of this .

[Laskos]

The job of a chess engine eval is not to provide a value based upon inferior play of opponent, but an objective measure of the position (assuming best play as far as it knows), at least in my eyes. Now if someone wants to change that definition, they are certainly free to do so, but I doubt many will find their idiosyncrasies particularly valuable. However, I suspect you already know all of this .

Well, jokes aside, we are back to "common sense" evals of most engines until now as evaluated in some sort of "pawns" in that 1, 3, 3, 5, 9.5 or similar values. With such evals, performance curve as a function of eval most often decays exponentially with eval (logistic tails), and not hyperbolically (atan tails). Also, the general calibration of close to center (0.00) values is off in Leela. Where from they got such unintuitive eval shape?

[jhellis3]

Where from they got such unintuitive eval shape?

Nowhere? The conversion function is side and color indifferent, as I am sure you already know.... The underlying scores which are converted to centi-pawns are purely a product of the NN score estimation which is purely a product of its self-play training data. So any "bias" which lc0 has stems directly from actual training game results. That a human may not find the results of its self play games intuitive is not part of its scoring criteria AFIAK.

[lkaufman]

The display in pawn units should be no problem for Lc0 or any other engine. A score of 3.00 for example means that the win probability is equal to what a neutral position with three extra pawns would get. Of course defining "neutral position" is tricky, some of us mean on a full board, some mean in a typical endgame, or an average over the whole game. There is no "right" answer, but it should be good enough to just remove pawns randomly from the opening position and set the display to show 2 pawns (for example) when two random pawns are gone from the start position. Probably the logistic function is close enough to get this about right.

[hgm]

It also depends on how the neural netowrk is trained. Learning it the win probability is probably not optimal, because then the progress measure once it has an advantage large enough to make the win certain will drown in the noise. This is actually a problem in Deus X obvious enough that I heard people complain about it. The function to translate back the NN output to a material measure should reflect the training of the latter.

[Uri Blass]

The display in pawn units should be no problem for Lc0 or any other engine. A score of 3.00 for example means that the win probability is equal to what a neutral position with three extra pawns would get. Of course defining "neutral position" is tricky, some of us mean on a full board, some mean in a typical endgame, or an average over the whole game. There is no "right" answer, but it should be good enough to just remove pawns randomly from the opening position and set the display to show 2 pawns (for example) when two random pawns are gone from the start position. Probably the logistic function is close enough to get this about right.

probability is dependent on the time control and the strength of the players.

The probability of black to win when white play without knight b1 is simply 100% if the players are strong enough and in this case if you work based on probabilities there is no difference between removing the knight and removing the queen.

[lkaufman]

But this is not what is happening with Lc0. It apparently assigns a win prob. for knight odds of well under 100% (I don't know the exact number, but I think it is probably something in the ballpark of 97%), but then converts that to a ridiculous pawn score. Your point only matters if the engine actually believes that the win prob. is 100% (or so close to it that we can call it 100%). As for other issues having to do with how the win prob. is determined, I don't think that matters here, I'm just arguing for consistency; if a typical position with an extra two pawns scores x percent, then x percent should translate back to two pawns.

[jhellis3]

Well, sounds like you better work on Komodo TB scores... . You are essentially criticizing a high score by LC0 because it knows the position is not just a knight up but more likely completely winning. This is similar to what one sees traditional engines with special endgame cases (TBs or known win code). Criticizing LC0 for having a superior eval (more quickly recognizing known wins) seems a bit silly, especially when K itself reports scores of +250 for TB wins where it may only be a pawn up....

Or if you want an objective measure, have Komodo play white in 100 games (different openings) against LC0 a knight down.

[lkaufman]

Well, sounds like you better work on Komodo TB scores... . You are essentially criticizing a high score by LC0 because it knows the position is not just a knight up but more likely completely winning. This is similar to what one sees traditional engines with special endgame cases (TBs or known win code). Criticizing LC0 for having a superior eval (more quickly recognizing known wins) seems a bit silly, especially when K itself reports scores of +250 for TB wins where it may only be a pawn up....

Or if you want an objective measure, have Komodo play white in 100 games (different openings) against LC0 a knight down.

But this is not the situation. For example if I remove a black knight (b8) instead of a white knight, the score goes to 63 pawns. Or if I remove b1 knight and c7 pawn, the score is 30 pawns. Lc0 understands that b8 odds is more winning than b1 odds, which is more winning than b8 for c7 odds, and the relative evals are reasonable. But they are about 14 times as much as they should be. Apparently the b8 knight score of 63 pawns appears to be about the maximum as removing more pieces makes the score decline from 63. I think it is fairly common for Lc0 to report scores in the balllpark of 10 pawns for positions where it cannot demonstrate a win against itself. This is just silly.