What is a backward pawn?

[Lyudmil Tsvetkov]

Chess engines are notoriously weak when playing with backward pawns, they simply do not understand them, even the strongest ones. As I am looking through engine games, I almost did not see a single game where one engine would use enemy backward pawns to secure a win, do you understand, even a single game. I ask myself what is the reason for this, and tried to look up official definitions from some sites.

Wikipedia says the following: In chess, a backward pawn is a pawn that is behind all pawns of the same color on the adjacent files and cannot be safely advanced.

And Chess Programming wiki has this one: A backward pawn is a pawn no longer defensible by own pawns and whose stop square is controlled by an opponent pawn or even by two pawns, the so called sentries. Thus, don't considering piece tactics, the backward pawn is not able to push forward without being lost, even establishing an opponent passer. If two opposing pawns on adjacent files in knight distance are mutually backward, the more advanced is not considered backward.

To try to clarify myself, I looked even at SF code and my confusion only grew, as I did not quite undertsand what pawns exactly SF considers backward, maybe my fault.

For me, the 2 official definitions above are very imperfect and insufficient. So I will give my definition for a backward pawn, and maybe people will comment. For me, a backward pawn is one that fulfills the following 5 conditions:

1. is the base pawn of a chain, in the usual case, or a chain pawn with some additional conditions
2. its front/advance/stop square is attacked by an enemy pawn
3. the abovementioned enemy pawn attacking the stop square is blocked by a friendly pawn
4. the base pawn of the chain or the chain pawn that would qualify for backward are necessarily not opposed
5. there are no friendly pawns on the same rank or behind the base or chain pawn on adjacent files, except when such pawns are also backward, or blocked by enemy pawns

I think this is a good and exhaustive definition. A definition that would allow to give penalties for backward pawns through ranks 7-4.

[d]6k1/1p5p/p5pP/P1p2p2/3p1P2/3P4/8/6K1 w - - 0 1
3 black backward pawns:
b7 is backward
c5 is backward because the only pawn that can support it behind, b7, is also backward
g6 is backward chain pawn because the only pawn on adjacent file that could support it, h7, is blocked

In all 3 cases you see the backward pawns are necessarily not opposed. That is a most essential requirement, as the main feature of backward pawns is that 1 enemy pawn stops the advance of 2 friendly pawns at the same time, by attacking the stop square of one of them, and blocking the other. That is very efficient in terms of resources, and very positional.

b7 is backward on the 7th rank, c5 is backward on the 5th rank, g6 is backward on the 6th rank.

[d]6k1/5p2/4p1p1/3p2P1/1p1P4/p7/P7/6K1 w - - 0 1
b4 is backward on the 4th rank, again not opposed and base of a chain

f7 is backward because of g5, not ooposed and base of a chain

e6 is backward on the 6th rank, chain pawn, not opposed and backward because the only pawn that could support it to advance, f7, is also backward

So that backwardness is fully related with the inability to advance.

[d]6k1/3p4/1p6/p3p1p1/P3P2p/1P1P3P/6P1/6K1 w - - 0 1
No backward pawns here, I do not see, do you?

d7 and g2 get some penalty usually for being weak pawns on the 7th rank that are underdeveloped, but that is quite different from normal backward pawns that are necessarily not opposed.
Does it make sense to consider b3 and b6 as backward, as they obviously cancel each other. Once again, opposed pawns can not be backward.
Does it make sense to consider d3 as backward, where is the efficiency and positionality here, as 2 pawns stop another 2 pawns? d3 is not backward, as it is opposed.
Does it make sense to consider g5 as backward? Of course not, as 2 pawns stop another 2. So that a prerequisite for a backward pawn is that 1 enemy pawns stops 2 friendly ones. Here you already see the efficiency.

I think, when you consider ranks with the abovementioned definition, you could score ranks 7 through 4, with 7th rank getting the biggest penalty, 6th rank 1/2 less, 5th even less, and 4th rank very small penalty lower than 5th. If you consider also opposed backward pawns, you could score only 2 ranks, 7th and 5th, with the 6th being a nonsense. That is why opposed definition for backward does not work.

What do you think of the above definition?
Do you have a better one?
What is the definition for backward in your own engine?
And why do you think engines completely misunderstand backward pawns?

[Vinvin]

Last time I checked the definition of "backward pawn", an "isolated pawn" that was not a "passed pawn" is backward too ...

[Ferdy]

[d]6k1/3p4/1p6/p3p1p1/P3P2p/1P1P3P/6P1/6K1 w - - 0 1
I consider b3, d3 and g2 as backward for white. The engine will strive to remove this bacward pawn by supporting the push by pieces. For example white will play b4, if axb4 and a white piece takes b4 too, what we have now is a black backward pawn in b6, the white a4 pawn is not backward because it is already in the fourth rank. So black will prepare something if ever white will achieve the push, because if black will not take the white's b4 pawn push, white has the posibility now of either moving b4b5 grabbing space or b4xa5.

[Henk]

Skipper says a backward pawn is a weak pawn with at least one opposite colored neighbor pawn at rowNr + 2.

A weak pawn is a pawn with no neighbor pawns of the same color at rows <= rowNr.

But I removed all code that used these definitions for very bad testing showed that it did not give any profit.

[Lyudmil Tsvetkov]

I think 2 absolutely necessary conditions for a backward pawn are that:
- the pawn is not opposed
- and 1 enemy pawn stops 2 friendly ones

[d]6k1/3p4/1p4p1/4P2p/P6P/8/8/6K1 w - - 0 1

How is it possible that a4,b6,d7 and e5 are backward pawns? They are not opposed, it is true, but it is 1 vs 1, no efficiency here, no positionality. I think considering those pawns backward is just a waste of resources. There is nothing special about those pawns. g6 on the other hand, is a true backward pawn, it is not opposed and it is 2 vs 1, 1 enemy pawns stops 2 friendly ones. There is positionality here, there is efficiency.

Maybe imperfect definitions are one of the reasons all modern engines play very bad with backward pawns, actually this is the element they play worse of all, and that is still easily fixable.

I would be glad if some other authors post some output of their engines to tell how many backward pawns they see on the positions posted.

I think there is much to be gained by implementing backward pawns according to correct definitions, even in top engines - 20,30, 50 elo or so. The reason is that backward pawns are extremely frequent, the most frequent pawn feature, maybe 5 times more frequent than passers, but if engines do not recognize them correctly, there is certainly nothing to be gained.

[Lyudmil Tsvetkov]

Last time I checked the definition of "backward pawn", an "isolated pawn" that was not a "passed pawn" is backward too ...

[d]6k1/8/2p5/8/p7/2P5/1P6/6K1 w - - 0 1
You want to say a4 and c6 are backward too?

[BeyondCritics]

Hi all,

Both of you miss this point: The definition of "backward pawn" in a chess engine is an extremly bold definition designed solely to be awarded a single, hefty penalty later.

Distinguish carefully:
"Backward pawn" for a chess player <-> "Backward pawn" for a chess engine.

This is a main problem in this forum: Chess knowledge representation is inadvertly mixed with chess engine logic.

The successful chess programmer is something of a hacker a this point. He expects something mathematically sound, that works for any random position his engine may encounter. Any exception to the rule could turn into a bug.

The succesful chess player on the other hand is mainly concerned about his intuition. He sees one position and then believes his has seen them all. Small exceptions to the rule are accepted as normal.

So please keep in mind the different mindsets of chess programmers and chess players.

To get a feeling for a working world class chess evaluation consider:
http://www.top-5000.nl/ZW_Rybka_Fruit.pdf

You will see that the evaluation is extremly limited. There are very few surviving concepts.

If you want to improve on that, i think the following principles should be adhered too at least:

A feature must have low volatility. Think about it: All positions with the same feature get the same penalty. If you award innocent backward pawns the same penalty than really backward pawns you asking for trouble. Both of you consider some innocent pawn structures, mixing them up with backward pawns. Big blunder!

Features must be orthogonal to each other. If you work on backward pawns, you must think about your other pawn features too. Overlap should be minimal. If you dont have othogonal features, your parameter estimates will later fight each other, emitting lot of noise. This error can make linear parameter estimation fail!

Concluding:
Interesting work on the semantic of backward pawns, but the logic is currently flawed.

Sorry, i currently dont have time to work on this positively. Maybe later.

Oliver

[Lyudmil Tsvetkov]

[d]6k1/3p4/1p6/p3p1p1/P3P2p/1P1P3P/6P1/6K1 w - - 0 1
I consider b3, d3 and g2 as backward for white. The engine will strive to remove this bacward pawn by supporting the push by pieces. For example white will play b4, if axb4 and a white piece takes b4 too, what we have now is a black backward pawn in b6, the white a4 pawn is not backward because it is already in the fourth rank. So black will prepare something if ever white will achieve the push, because if black will not take the white's b4 pawn push, white has the posibility now of either moving b4b5 grabbing space or b4xa5.

[d]6k1/3p4/1p6/4p1p1/P3P2p/3P3P/6P1/6K1 w - - 0 1
So why on Earth b6 should be backward and a4 not? Those pawns already get different psqt values for 4th and 3rd rank respectively, but apart from that, what is the difference between them? They both can not advance, it is 1 vs 1, some bonus for pawn control only for a4 is OK, but that would be 5-8cps, we are talking here of giving 20-30cps and more penalty for a real backward pawn. Is it possible that b6 gets penalised so much?

And also why d3 is backward? It can not advance, but black's d pawn also can not advance past the 6th rank, and it is an equal distribution of resources, 2 vs 2, neither side gets anything special from that situation. So that I think, although d3 could be penalised, this is wrong in practical terms.

To ask you one question, Ferdinand, how is Deuterium able to play so strong when it understands backward pawns so poorly?

[Lyudmil Tsvetkov]

Hi all,

Both of you miss this point: The definition of "backward pawn" in a chess engine is an extremly bold definition designed solely to be awarded a single, hefty penalty later.

Distinguish carefully:
"Backward pawn" for a chess player <-> "Backward pawn" for a chess engine.

This is a main problem in this forum: Chess knowledge representation is inadvertly mixed with chess engine logic.

The successful chess programmer is something of a hacker a this point. He expects something mathematically sound, that works for any random position his engine may encounter. Any exception to the rule could turn into a bug.

The succesful chess player on the other hand is mainly concerned about his intuition. He sees one position and then believes his has seen them all. Small exceptions to the rule are accepted as normal.

So please keep in mind the different mindsets of chess programmers and chess players.

To get a feeling for a working world class chess evaluation consider:
http://www.top-5000.nl/ZW_Rybka_Fruit.pdf

You will see that the evaluation is extremly limited. There are very few surviving concepts.

If you want to improve on that, i think the following principles should be adhered too at least:

A feature must have low volatility. Think about it: All positions with the same feature get the same penalty. If you award innocent backward pawns the same penalty than really backward pawns you asking for trouble. Both of you consider some innocent pawn structures, mixing them up with backward pawns. Big blunder!

Features must be orthogonal to each other. If you work on backward pawns, you must think about your other pawn features too. Overlap should be minimal. If you dont have othogonal features, your parameter estimates will later fight each other, emitting lot of noise. This error can make linear parameter estimation fail!

Concluding:
Interesting work on the semantic of backward pawns, but the logic is currently flawed.

Sorry, i currently dont have time to work on this positively. Maybe later.

Oliver

2 tuned wrong features are orthogonal, but 2 right features that are not tuned are not othogonal. Still, if you tune the 2 right features, you will beat the 2 wrong features on orthogonality.

Rybka/Fruit eval used to be world-class, now it is not. Now even SF has better eval. Thinking that fewer features work better is a wrong approach. You need as little features as possible, but you can not do without the important ones. Backward pawns is an extremely important feature. And guess what? Engines are completely blind for backward pawns, very weak, even Houdini, SF and Komodo.

That is what we are talking about, we should have the right definitions, and as it seems, engines in general, even the top, lack the right definition and penalisation for backward pawns. And that is why they are unable to play with them. They understand doubled pawns to a sufficient degree, isolated pawns almost very well, candidates, passers extremely well, because a lot of work has gone there, but not backward pawns, that is an exception. That is why I do not think I am wrong when I say 'Please, take a look at your definitions'. Engines do not understand backward pawns.

[d]6k1/5p2/4p1p1/3p2P1/1p1P4/p7/P7/6K1 w - - 0 1
OK, how many engines would see e6 as backward above, or b4? And what would be the penalty they would apply to those pawns?

Specifically e6 is a major feature of the position, it can not simply be left without a backward penalty.

Interesting if some programmers could share for how many ranks they apply the backward penalty? And is it uniform?

[bob]

My take on this is that you are somewhat mixing terms.

A pawn that is defended by a pawn is not considered backward in any book I have (g6 in your first example as it is defended by h7). The fact that it can not safely advance is not so important when it is perfectly safe where it is. If you want to talk about it being immobile, that is a different issue.

the "unopposed" requirement stems from the rook, in the books I have read over the years. A pawn that is backward (can not be safely defended by a friendly pawn) is generally considered to be worse if it is (a) on a half-open file (no enemy pawn in front of it AND (b) the enemy has rooks to attack it.

IMHO, a pawn that is backward and unopposed is not any weaker or stronger than a backward pawn that is opposed, if there are no enemy rooks to contend with. Any backward pawn is weak, and inviting to enemy kings in the endgame. backward pawns on half-open files make inviting targets even in the middle game if you have rooks to attack on the file.

I've seen games decided by two types of backward pawns, repeatedly.

The first is the classic backward pawn on a half-open file, which is just as bad as an isolated pawn on a half-open file, when the enemy has rooks. Pile up on the pawn, tying up the opponent by making him defend, and either win the pawn outright, or switch to some other idea once he is tied down and not coordinated very well.

The second is a pawn that is backward, but the file is not half-open. If you have just one, you might manage to defend it with your king, depending on the position. If you have two, the enemy king can make you commit to saving one and then to eat the other pawn chain. Doesn't matter whether the file is half open or not.