Knight equals 48 pawns?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Nay Lin Tun
Posts: 599
Joined: Mon Jan 16, 2012 5:34 am

Re: Knight equals 48 pawns?

Post by Nay Lin Tun » Sat Feb 23, 2019 10:56 pm

lkaufman wrote:
Sat Feb 23, 2019 4:52 pm
jp wrote:
Sat Feb 23, 2019 2:13 am
lkaufman wrote:
Fri Feb 22, 2019 6:45 pm
By the way, I note that it is not feasible to play against Lc0 at knight odds, because it soon starts to blunder pieces almost randomly. At smaller handicaps like pawn f7 and move or two this doesn't happen, it plays very well. It's almost as if all win probabilities beyond 99% are the same.
Does it do this blundering from both sides (knight up and knight down) or only from knight up?
Both sides, but more noticeably when down a piece or more. Basically, if its winning prob. is below 2% or so, it just "resigns" by making random blunders. Even taking more time doesn't generally cure this. Although Lc0 can probably beat top GMs giving them pawn or maybe pawn and move handicap just like Komodo does, I don't think that Lc0 can even beat a 1400 rated human at knight odds (Komodo can give knight odds in blitz to GMs, but in slow games only to players aroun 2000). Even with somewhat higher win probs., say around 5% or so, Lc0 still blunders, though not as often. I've seen the same behavior from a couple different networks, I don't think the specific network is the problem. Perhaps it's a problem with rarely occurring material balances, such as being down two pieces or more.
Knight Odds of Lco were already played by 2900 blitz rated players like Daniel Nardoitsky and other GMs at blitz time control. Lc0 won them comfortably. Beat them with 70-80% score with knight odd.
please review the games on his account by yourself.

https://lichess.org/blog/XBsCBBMAACUA3C ... chess-zero

lkaufman
Posts: 4222
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Knight equals 48 pawns?

Post by lkaufman » Sun Feb 24, 2019 2:56 am

Nay Lin Tun wrote:
Sat Feb 23, 2019 10:56 pm
lkaufman wrote:
Sat Feb 23, 2019 4:52 pm
jp wrote:
Sat Feb 23, 2019 2:13 am
lkaufman wrote:
Fri Feb 22, 2019 6:45 pm
By the way, I note that it is not feasible to play against Lc0 at knight odds, because it soon starts to blunder pieces almost randomly. At smaller handicaps like pawn f7 and move or two this doesn't happen, it plays very well. It's almost as if all win probabilities beyond 99% are the same.
Does it do this blundering from both sides (knight up and knight down) or only from knight up?
Both sides, but more noticeably when down a piece or more. Basically, if its winning prob. is below 2% or so, it just "resigns" by making random blunders. Even taking more time doesn't generally cure this. Although Lc0 can probably beat top GMs giving them pawn or maybe pawn and move handicap just like Komodo does, I don't think that Lc0 can even beat a 1400 rated human at knight odds (Komodo can give knight odds in blitz to GMs, but in slow games only to players aroun 2000). Even with somewhat higher win probs., say around 5% or so, Lc0 still blunders, though not as often. I've seen the same behavior from a couple different networks, I don't think the specific network is the problem. Perhaps it's a problem with rarely occurring material balances, such as being down two pieces or more.
Knight Odds of Lco were already played by 2900 blitz rated players like Daniel Nardoitsky and other GMs at blitz time control. Lc0 won them comfortably. Beat them with 70-80% score with knight odd.
please review the games on his account by yourself.

https://lichess.org/blog/XBsCBBMAACUA3C ... chess-zero
Thanks, but I couldn't find either the games or the results of the games from that link, but I saw a reference to a "modified version of Leela" for the games. Can you give a link to the actual results or else summarize them? Something isn't right here; the two recent networks I downloaded (including the one with the highest estimated CCRL rating) both just gave away pieces when giving knight odds as if they were playing Losing chess; yet there is nothing wrong with my setup since they play well enough to beat me at f7 and two or even three move odds, about like Komodo does. My guess is that they found a way to make it play good chess despite the knight odds for this "modified version"; does anyone know anything about this?
Komodo rules!

lkaufman
Posts: 4222
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Knight equals 48 pawns?

Post by lkaufman » Sun Feb 24, 2019 5:18 am

Nay Lin Tun wrote:
Sat Feb 23, 2019 10:56 pm
lkaufman wrote:
Sat Feb 23, 2019 4:52 pm
jp wrote:
Sat Feb 23, 2019 2:13 am
lkaufman wrote:
Fri Feb 22, 2019 6:45 pm
By the way, I note that it is not feasible to play against Lc0 at knight odds, because it soon starts to blunder pieces almost randomly. At smaller handicaps like pawn f7 and move or two this doesn't happen, it plays very well. It's almost as if all win probabilities beyond 99% are the same.
Does it do this blundering from both sides (knight up and knight down) or only from knight up?
Both sides, but more noticeably when down a piece or more. Basically, if its winning prob. is below 2% or so, it just "resigns" by making random blunders. Even taking more time doesn't generally cure this. Although Lc0 can probably beat top GMs giving them pawn or maybe pawn and move handicap just like Komodo does, I don't think that Lc0 can even beat a 1400 rated human at knight odds (Komodo can give knight odds in blitz to GMs, but in slow games only to players aroun 2000). Even with somewhat higher win probs., say around 5% or so, Lc0 still blunders, though not as often. I've seen the same behavior from a couple different networks, I don't think the specific network is the problem. Perhaps it's a problem with rarely occurring material balances, such as being down two pieces or more.
Knight Odds of Lco were already played by 2900 blitz rated players like Daniel Nardoitsky and other GMs at blitz time control. Lc0 won them comfortably. Beat them with 70-80% score with knight odd.
please review the games on his account by yourself.

https://lichess.org/blog/XBsCBBMAACUA3C ... chess-zero
I just played the following game against the supposedly best net of Lc0, at 5' + 5", GPU 1060, knight (b1) odds. 1.Nf3 d5 2.e3 Nf6 3.d4 Bg4 4.Ne5??? and I took the queen. Using infinite analysis it still prefers 4.Ne5??? even after a full minute, so a faster GPU or longer TC wouldn't have helped. And after I capture the queen it doesn't even recapture but still thinks it has a 3.6% win prob!! But at two pawn and move odds (b7 and c7) I see no bad moves, it plays well enough to beat a top GM in blitz I think. Are you sure that these games were at full knight odds (there is a mention of two pawns or knight odds)? Something had to be different, Naroditsky would not lose to an engine that hangs its queen on move 4. Maybe some parameter needs special setting for knight odds?
Komodo rules!

dkappe
Posts: 385
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: Knight equals 48 pawns?

Post by dkappe » Sun Feb 24, 2019 2:01 pm

Different nets react differently to unusual positions. The t30 nets used in this TCEC Season (like 32930) had a real allergy to double queen positions. The late t10 nets (11248 and 11258) were a little bit more robust in this regard. I believe most of the public odds games were played against t10 nets.

My recommendation would be to try 11248 or 11258 for these odds positions.

lkaufman
Posts: 4222
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Knight equals 48 pawns?

Post by lkaufman » Sun Feb 24, 2019 6:52 pm

You were quite correct, except for the "little bit more robust" comment; the difference is day and night. With the 11248 network the win probs are all much closer to 50%, so knight odds gets 13% instead of 1%! Of course 13% for knight odds is ridiculous, but the result is that it plays quite well giving knight odds, and I lost the first blitz game I tried with it. It seems that with this version the silly play and absurd relative win probs only show up when it is down about two knights or more or the equivalent, instead of just one knight, so for any games against strong human players this does solve the problem. I note that this version (well actually the other one you mentioned, 11258) is listed as having an estimated CCRL rating just around 25 elo below the top one, so for handicap play it is a clear choice, probably even for smaller handicaps like pawn and move.
Komodo rules!

User avatar
hgm
Posts: 24602
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Knight equals 48 pawns?

Post by hgm » Sun Feb 24, 2019 7:03 pm

This is a very good illustration of how bad it is to have the score by which an engine plays saturate for a forcible win, or put at zero for a theoretical draw. It works the other way as well: the engine will not only stop offering meaningful resistance in a lost position; it will also not try to convert wins. Training the network for expected distance to win would solve this problem.

Jesse Gersenson
Posts: 582
Joined: Sat Aug 20, 2011 7:43 am
Contact:

Re: Knight equals 48 pawns?

Post by Jesse Gersenson » Tue Feb 26, 2019 10:46 am

Lc0 misevaluates the starting position of knight odds. Lc0 plays chess very well.

With a little more work, one could make a proof showing the evaluation of odds starting positions are poor indicators of a NN's ability to evaulate standard chess positions.

User avatar
hgm
Posts: 24602
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Knight equals 48 pawns?

Post by hgm » Wed Feb 27, 2019 12:41 pm

The poit is that if it cannot evaluate an odds start position, it will also not be able to evaluate other positions with a material imbalance. And this indeed shows: Leela type engines are already infamous for the excruciatingly slow way they convert wins.

Post Reply