Lyudmil Tsvetkov wrote:
Maybe it is also time to bet with Arjun that an uniform 60cps bonus for a blocked pawn on the 6th rank, excluding only h6 that already has a big bonus, will pass both short and LTC. I.e., a uniform bonus only for blocked pawns on the 6th, excluding the 5th rank, where term clashes are bigger and chances to go astray increase.
Ok, I have pushed two quick tests to the framework to see if it is any improvement. 60 seems very high, so I have also pushed 40.
Thanks Arjun.
You are my biggest benefactor and possibly grave-digger. (or do they say undertaker?)
Concerning values, Stefan's 260 was even more shocking, but you have to consider that 60cps SF values is some 30cps standard ones, so that 30cps standard bonus for 6th rank is not big at all. The question is if it will still clash with something...apart from the already assigned bonus for edge h6/a6 storming pawn, which should be negligeable overall. (but excluding just the h6/a6 storming pawns from this would be even more consistent). I would be very grateful if you leave at least one of the tests for STC regardless of how the values score at 5 seconds. You know, those pawns scale differently, you have the bitter storm experience.
Many thanks again, Arjun!
Hi Lyudmil, first note that SF eval has been scaled down in recent times so it is now in line with other engines like Houdini and Komodo. You might have noticed this in the TCEC games of stage 3.
In either case, the evaluation score you see in the GUI is different from the internal values that are juggled around. The external score is scaled down in notation.cpp by this line:
where v is the final value of the internal evaluation
and PawnValueEg = 258.
I tried excluding the storm values from it, but in local tests, this did quite poorly. For some reason, it works better the way I have submitted it. (Fingers crossed).
We will see how this goes first. I will reschedule for 15" later.
[d]r1bk1b1r/ppp2ppp/2p5/4Pn2/8/5N2/PPP2PPP/RNB2RK1 w - - 0 9
Well, this is certainly not an endgame. But the queens are off. Rybka, very realistically, evaluates this main Berlin position with 15cps white edge, SF sees 80cps!! white advantage. I am not a theoretician at all, but people, Kramnik among them, say that the Berlin is supposed to be a very sound opening, drawish at that, so that those whopping 80cps really amaze. Would be glad if anyone proves that latest development behaves otherwise.
[d]r1b1kb1r/ppp2ppp/2p5/4Pn2/8/2N2N2/PPP2PPP/R1BR2K1 b - - 0 10
One of the drawish lines includes black playing Bd7, Be7 and later exchanging the dark-coloured bishop for the white knight on g5, so that an opposite-coloured bishops ending arises. SF quite probably does not choose this line, as it does not scale score properly, so that it believes the opposite-coloured bishops ending with white pawn majority on the king side and doubled black pawns on the queen side is lost for black. Wrongly at that, because the opposite-coloured bishops hold.
[d]2k4r/p1p1b1p1/1pn1p2p/2p1P3/4NPP1/2P1B1K1/PP5P/3R4 w - - 0 23
Instead of an opposite-coloured bishop ending, SF enters same-coloured bishops ending, with the same pawn characteristics and probably some minor differences. Here Rybka gives 60cps white edge, SF a full pawn! Now, it is true that SF did not lose the game, probably because Rybka went wrong somewhere, but frankly, why should SF suffer so much here, when it could easily have chosen the much better opposite-coloured bishops line?
[d]7r/2pkb3/ppn1p1p1/2p1P3/4NPKP/1PP5/P4B2/6R1 w - - 0 32
I will not comment, SF saves this somehow, but this is suffering.
Do you believe SF plays well the Berlin?
Maybe someone will post other examples of SF in the Berlin, would be a nice contribution.
[d]r3k2r/pppb1pp1/2p4p/4Pn2/5B2/2N5/PPP2PPP/R2R2K1 b - - 0 14
Interesting to know how SF evaluates this above, should be pretty much drawish. Or maybe someone will post a more relevant Berlin position.
In any case, this should be a good position to judge necessity and percentage of scaling in a relatively rich position without queens on the midground between middlegame and endgame. Quite possibly, in positions without queens, percentage of scaling must be much bolder, but who knows, you need a lot of testing.
Lyudmil Tsvetkov wrote:
Maybe it is also time to bet with Arjun that an uniform 60cps bonus for a blocked pawn on the 6th rank, excluding only h6 that already has a big bonus, will pass both short and LTC. I.e., a uniform bonus only for blocked pawns on the 6th, excluding the 5th rank, where term clashes are bigger and chances to go astray increase.
Ok, I have pushed two quick tests to the framework to see if it is any improvement. 60 seems very high, so I have also pushed 40.
Thanks Arjun.
You are my biggest benefactor and possibly grave-digger. (or do they say undertaker?)
Concerning values, Stefan's 260 was even more shocking, but you have to consider that 60cps SF values is some 30cps standard ones, so that 30cps standard bonus for 6th rank is not big at all. The question is if it will still clash with something...apart from the already assigned bonus for edge h6/a6 storming pawn, which should be negligeable overall. (but excluding just the h6/a6 storming pawns from this would be even more consistent). I would be very grateful if you leave at least one of the tests for STC regardless of how the values score at 5 seconds. You know, those pawns scale differently, you have the bitter storm experience.
Many thanks again, Arjun!
Hi Lyudmil, first note that SF eval has been scaled down in recent times so it is now in line with other engines like Houdini and Komodo. You might have noticed this in the TCEC games of stage 3.
In either case, the evaluation score you see in the GUI is different from the internal values that are juggled around. The external score is scaled down in notation.cpp by this line:
where v is the final value of the internal evaluation
and PawnValueEg = 258.
I tried excluding the storm values from it, but in local tests, this did quite poorly. For some reason, it works better the way I have submitted it. (Fingers crossed).
We will see how this goes first. I will reschedule for 15" later.
Now Arjun, this is a bit murky for me, let us see how the test goes...
Again, the paradox, but also the reason why bolder scaling might not work in tests, is that it is safe to scale this one:
[d]rnb1k1nr/pppp1ppp/8/8/8/8/PPPPPPPP/RNB1K1NR w KQkq - 0 1
but not this one:
[d]2bqk3/pppp1ppp/8/8/8/8/PPPPPPPP/2BQK3 w - - 0 1
Indeed, how safe it is to scale this:
[d]6k1/1p2qpp1/p3b2p/8/8/P1BQP3/1P3PPP/6K1 w - - 0 1
From that perspective, at least for my current limited understanding, the decision if to scale and how much depends very much on total material, as you scale more with less material, but even more so on presence of queens on the board. I would never scale with queens on, but maybe a limited scaling is warranted even in this case. What is certain however, is that queenless positions, middlegame or endgame, could be scaled much bolder.
I do not know if this is the right thread, but I will ask here.
You know, I am very curious, and that is a bad thing.
Still, below the data from the last 3 standard and SMP (3 threads) SF regression tests:
27.02.2014 standard 39.25 elo SMP 37.55 difference 1.70 elo
25.03.2014 standard 43.70 elo SMP 39.24 difference 4.46 elo
26.04.2014 standard 57.08 elo SMP 51.04 difference 6.04 elo
As you see, the difference between standard single core and SMP tests continuously rises, with some peculiar behaviour. What I observe?
On 27.02, at 40 elo increase over SF DD, SF loses only 1.7 elo with SMP; on 26.04, at almost 60 elo increase over DD, 1/3 of the overall increase, SF already loses 6 elo with SMP; from 27.02 to 25.03 SF adds 4.5 elo and loses 2.76 elo at the same time with SMP, obviously the most macabre scaling period. From 25.03 to 26.04 SF adds 13.40 elo standard and loses 1.60 elo with SMP.
So, basically, the SMP performance on 27.02 is 3 times better than the SMP performance from 25.03 to 26.04. The SMP performance from 27.02 to 25.03 is absolutely disastrous for some reason.
Bearing in mind that SMP is very much tantamount to scalability, closely related to performance at longer TC, it is clear that the changes introduced since DD until 27.02 scale much better than the changes introduced in the last month, and even more so than the changes introduced from February till end of March.
How do you explain this peculiar behaviour? What were the changes that scaled so well in the beginning, and what are the changes that scale considerably less so now? What were the changes from February till March that scaled so extremely bad? Could we learn a lesson at looking at the history page what change it is good to make and what not, so that the scalability of SF is optimal?
Any comments very much appreciated. I am new and weak in this branch so maybe people will elucidate me. At the end of the day, we want an engine that scales best and plays best at longer TC and multiple threads as the TCEC conditions.
Lyudmil Tsvetkov wrote:I do not know if this is the right thread, but I will ask here.
You know, I am very curious, and that is a bad thing.
Still, below the data from the last 3 standard and SMP (3 threads) SF regression tests:
27.02.2014 standard 39.25 elo SMP 37.55 difference 1.70 elo
25.03.2014 standard 43.70 elo SMP 39.24 difference 4.46 elo
26.04.2014 standard 57.08 elo SMP 51.04 difference 6.04 elo
As you see, the difference between standard single core and SMP tests continuously rises, with some peculiar behaviour. What I observe?
On 27.02, at 40 elo increase over SF DD, SF loses only 1.7 elo with SMP; on 26.04, at almost 60 elo increase over DD, 1/3 of the overall increase, SF already loses 6 elo with SMP; from 27.02 to 25.03 SF adds 4.5 elo and loses 2.76 elo at the same time with SMP, obviously the most macabre scaling period. From 25.03 to 26.04 SF adds 13.40 elo standard and loses 1.60 elo with SMP.
So, basically, the SMP performance on 27.02 is 3 times better than the SMP performance from 25.03 to 26.04. The SMP performance from 27.02 to 25.03 is absolutely disastrous for some reason.
Bearing in mind that SMP is very much tantamount to scalability, closely related to performance at longer TC, it is clear that the changes introduced since DD until 27.02 scale much better than the changes introduced in the last month, and even more so than the changes introduced from February till end of March.
How do you explain this peculiar behaviour? What were the changes that scaled so well in the beginning, and what are the changes that scale considerably less so now? What were the changes from February till March that scaled so extremely bad? Could we learn a lesson at looking at the history page what change it is good to make and what not, so that the scalability of SF is optimal?
Any comments very much appreciated. I am new and weak in this branch so maybe people will elucidate me. At the end of the day, we want an engine that scales best and plays best at longer TC and multiple threads as the TCEC conditions.
That is not correct.
When TC/depth increases, draw-ratio increases. Elo calculation takes into account draw-ratio, so naturally as SMP games tend to have deeper depths, elo gain is lower because there are more draws. (If you see carefully, win-loss ratio remains quite consistent).
Also, as elo difference between engines increases (for eg. SF DD vs master is now ~60 elo), the elo curve starts flattening out slightly. It flattens out even more for SMP. It is a normal thing. Nothing to worry about.
That is why the relative strength of engines at TCEC time control (with 16-cores) is much closer than on rating lists at bullet/blitz TC on standard hardware. The draw-ratio is orders of multitude higher in TCEC than in say LSratinglist. That's why you have so many close encounters even between engines that are 100-200 elo apart on standard rating lists. As TC/depth increases, elo difference decreases.
Last edited by arjuntemurnikar on Fri May 02, 2014 3:21 am, edited 1 time in total.
OK, some statistics from the history page, mentally done:
from DD to 27.02 there are 25 successful patches, 10 eval, 8 search, 2 time, one speed optimization, and 4 I do not know what, mainly Uri patches. Of those, 21 patches were tested and passed with SPRT 0;6, 1 patch passed SPRT -3;1 and 4 patches, mainly Uri simplifications, passed SPRT -4;0. Overall, 80% passed SPRT 0;6.
from 27.02 to 25.03 there are 4 successful patches, 1 eval, 1 search (restore old window size), one technical (TT replace) and 1 contempt. Of those, 2 patches passed SPRT 0;6 and 2 SPRT -3;1. Overall, 50% passed SPRT 0;6.
from 25.03 to 26.04 there are 17 successful patches, almost all of them eval, plus 3 or 4 contempt patches. Of those 17, 9 passed SPRT -3;1 and 8 SPRT 0;6. Overall, close to 50% passed SPRT 0;6.
Do you find this data of any interest and how do you comment on it?
My first impression is that eval patches since SF DD are twice or even more bigger number than search patches. (but search patches might have a bigger relative weight, I do not know)
Second impression is that the most successful period of scaling is linked to highest percentage of patches that passed SPRT 0;6, simplification patches almost did not exist then. The least successful period as well as the last period are linked to patches half of which passed SPRT -3;1, the simplification test. An obvious conclusion would be that SPRT -3;1, simplification tests, are conducive to scaling deficiency, the more you simplify, the worse the scaling of the engine with multiple threads, but also meaning with bigger time.
It is also notable that an almost equal number of successful patches in the first period contributed 3 times as many elo as the patches in the last period. Maybe it is because of search patches, maybe because of simplification.
Lyudmil Tsvetkov wrote:OK, some statistics from the history page, mentally done:
from DD to 27.02 there are 25 successful patches, 10 eval, 8 search, 2 time, one speed optimization, and 4 I do not know what, mainly Uri patches. Of those, 21 patches were tested and passed with SPRT 0;6, 1 patch passed SPRT -3;1 and 4 patches, mainly Uri simplifications, passed SPRT -4;0. Overall, 80% passed SPRT 0;6.
from 27.02 to 25.03 there are 4 successful patches, 1 eval, 1 search (restore old window size), one technical (TT replace) and 1 contempt. Of those, 2 patches passed SPRT 0;6 and 2 SPRT -3;1. Overall, 50% passed SPRT 0;6.
from 25.03 to 26.04 there are 17 successful patches, almost all of them eval, plus 3 or 4 contempt patches. Of those 17, 9 passed SPRT -3;1 and 8 SPRT 0;6. Overall, close to 50% passed SPRT 0;6.
Do you find this data of any interest and how do you comment on it?
My first impression is that eval patches since SF DD are twice or even more bigger number than search patches. (but search patches might have a bigger relative weight, I do not know)
Second impression is that the most successful period of scaling is linked to highest percentage of patches that passed SPRT 0;6, simplification patches almost did not exist then. The least successful period as well as the last period are linked to patches half of which passed SPRT -3;1, the simplification test. An obvious conclusion would be that SPRT -3;1, simplification tests, are conducive to scaling deficiency, the more you simplify, the worse the scaling of the engine with multiple threads, but also meaning with bigger time.
It is also notable that an almost equal number of successful patches in the first period contributed 3 times as many elo as the patches in the last period. Maybe it is because of search patches, maybe because of simplification.
And again, as explained in my earlier post, the apparent "regression" for SMP is normal, so your conclusions about SPRT -3, 1 are absolutely out of place. There is no reason to doubt the -3, 1 simplification approach because the statistics are sound.
And as for the frequency of eval patches vs search patches and simplification patches vs 0, 6 patches, it is completely irrelevant. Patches arise from ideas, and it is simply a random occurrence that in the early period say, lots of search related ideas were flowing around, and now lots of eval and simplification ideas are flowing around.
Lyudmil Tsvetkov wrote:OK, some statistics from the history page, mentally done:
from DD to 27.02 there are 25 successful patches, 10 eval, 8 search, 2 time, one speed optimization, and 4 I do not know what, mainly Uri patches. Of those, 21 patches were tested and passed with SPRT 0;6, 1 patch passed SPRT -3;1 and 4 patches, mainly Uri simplifications, passed SPRT -4;0. Overall, 80% passed SPRT 0;6.
from 27.02 to 25.03 there are 4 successful patches, 1 eval, 1 search (restore old window size), one technical (TT replace) and 1 contempt. Of those, 2 patches passed SPRT 0;6 and 2 SPRT -3;1. Overall, 50% passed SPRT 0;6.
from 25.03 to 26.04 there are 17 successful patches, almost all of them eval, plus 3 or 4 contempt patches. Of those 17, 9 passed SPRT -3;1 and 8 SPRT 0;6. Overall, close to 50% passed SPRT 0;6.
Do you find this data of any interest and how do you comment on it?
My first impression is that eval patches since SF DD are twice or even more bigger number than search patches. (but search patches might have a bigger relative weight, I do not know)
Second impression is that the most successful period of scaling is linked to highest percentage of patches that passed SPRT 0;6, simplification patches almost did not exist then. The least successful period as well as the last period are linked to patches half of which passed SPRT -3;1, the simplification test. An obvious conclusion would be that SPRT -3;1, simplification tests, are conducive to scaling deficiency, the more you simplify, the worse the scaling of the engine with multiple threads, but also meaning with bigger time.
It is also notable that an almost equal number of successful patches in the first period contributed 3 times as many elo as the patches in the last period. Maybe it is because of search patches, maybe because of simplification.
And again, as explained in my earlier post, the apparent "regression" for SMP is normal, so your conclusions about SPRT -3, 1 are absolutely out of place. There is no reason to doubt the -3, 1 simplification approach because the statistics are sound.
And as for the frequency of eval patches vs search patches and simplification patches vs 0, 6 patches, it is completely irrelevant. Patches arise from ideas, and it is simply a random occurrence that in the early period say, lots of search related ideas were flowing around, and now lots of eval and simplification ideas are flowing around.
You are seeing patterns where none exist.
There is a pattern, and it is very clear.
It is a pity that no one else wants to comment.
This is not a chance data, it is very consistent.
But this makes perfectly sense: with longer TC and computer power knowledge gains in importance as the relative speed loss from calculating that knowledge is already lower, it is bigger at shorter TC where resources are scarce. At the same time, with simplifications, a couple of million more or less computed lines in LTC and good hardware already do not matter, as randomness is already too big.
So that there is a pattern and it is clear. Knowledge scales well at LTC, removing knowledge scales bad at LTC and multiple threads. I hope SF gets the warning while it is still early. You can not simplify all the time, you need to add knowledge. But, how to do that, when people like their patches to succeed, and with simplification success chances are much higher. So basically everybody goes into simplification, while people should have been doing exactly the opposite: concentrate all their energy so that knowledge patches are implemented by consistent testing.
I really hope SF gets the signal while it is still early. You can imagine what happens with no-knowledge patches at 16 threads and 1 hour per game...
I.e., a uniform bonus only for blocked pawns on the 6th, excluding the 5th rank, where term clashes are bigger and chances to go astray increase.
