Laskos wrote: ↑Mon Dec 16, 2019 6:56 am
Second, Lc0 testing, between best t40 and t30 nets:
2000 games at 100 nodes/move:
+1064 =386 -550 62.8%
White score: 53.2%
1000 games at 1000 nodes/move
+514 =303 -183 66.5%
White score: 51.4%
Draw rate increases to longer TC, Armageddon score decreases from above towards 50%. Again, a promising result.
And again good result at 5000 nodes/move Lc0 testing between best t40 and t30 nets:
1000 games at 5000 nodes/move
+509 =366 -125 69.2%
White score: 50.9%
Nice progression from lower nodes count. AB engines have 48.3% White score at similar strength. All in all, very balanced and seems to slowly converge towards close to 50%.
I decided to check out your one bad result, Komodo self-play, to see if it got better or worse with more time and more threads. At 2' + 1" on four threads, latest Komodo dev. got 79 wins, 98 draws, and one loss for 44.4% Armageddon score, somewhat better than your result for selfplay. So at least it's moving in the right direction. With your results for non-selfplay averaging around 49.6% (using only the longest TC for Lc0), looks like this version is most likely as close as we can hope to get to being fair without crossing the line of White having a forced win (it seems). It's better to be below this line than above it, so I think this may be the perfect Armageddon variant at last! I'll try to get some more data and look for ways to promote the idea.
Some experiments with Komodo's Contempt. White Contempt = true in all case. Here are the result against Komodo Contempt = 0 of Contempt 25,50,75,100.
It is pretty sure Contempt = 50 performs significantly better than Contempt = 75 or all other contempts
Overall, White performance is 47.1%.
But Contempt = 75 against Contempt = 0 shows a White win performance at 44.5%, as in earlier tests. While the more adequate Contempt = 50 shows a White win performance of 50.2%. The difference in White performance in these two cases is significant. So, it might be that the low White win ratio of Contempt = 75 is partly due to an inadequately chosen value of the Contempt (aside identical Komodo engines in earlier tests)
Will check now self-play of Contempt = 50 Komodos.
It is pretty sure Contempt = 50 performs significantly better than Contempt = 75 or all other contempts
Overall, White performance is 47.1%.
But Contempt = 75 against Contempt = 0 shows a White win performance at 44.5%, as in earlier tests. While the more adequate Contempt = 50 shows a White win performance of 50.2%. The difference in White performance in these two cases is significant. So, it might be that the low White win ratio of Contempt = 75 is partly due to an inadequately chosen value of the Contempt (aside identical Komodo engines in earlier tests)
Will check now self-play of Contempt = 50 Komodos.
Tested also in self-play Contempt = 50 at 30 + 0.3s.
At Contempt = 75 the result was:
Komodo self-play (Contempt=75):
400 games at 30 + 0.3s:
+168 =203 -29 67.3%
White score: 42.0%
Now:
Komodo self-play (Contempt=50):
400 games at 30 + 0.3s:
+191 =183 -26 70.6%
White score: 47.8%
So, it seems fine with the adequate Contempt = 50 used. I am not sure that this Contempt is universal across time controls.
Laskos wrote: ↑Mon Dec 16, 2019 6:56 am
Second, Lc0 testing, between best t40 and t30 nets:
2000 games at 100 nodes/move:
+1064 =386 -550 62.8%
White score: 53.2%
1000 games at 1000 nodes/move
+514 =303 -183 66.5%
White score: 51.4%
Draw rate increases to longer TC, Armageddon score decreases from above towards 50%. Again, a promising result.
And again good result at 5000 nodes/move Lc0 testing between best t40 and t30 nets:
1000 games at 5000 nodes/move
+509 =366 -125 69.2%
White score: 50.9%
Nice progression from lower nodes count. AB engines have 48.3% White score at similar strength. All in all, very balanced and seems to slowly converge towards close to 50%.
I decided to check out your one bad result, Komodo self-play, to see if it got better or worse with more time and more threads. At 2' + 1" on four threads, latest Komodo dev. got 79 wins, 98 draws, and one loss for 44.4% Armageddon score, somewhat better than your result for selfplay. So at least it's moving in the right direction. With your results for non-selfplay averaging around 49.6% (using only the longest TC for Lc0), looks like this version is most likely as close as we can hope to get to being fair without crossing the line of White having a forced win (it seems). It's better to be below this line than above it, so I think this may be the perfect Armageddon variant at last! I'll try to get some more data and look for ways to promote the idea.
Some experiments with Komodo's Contempt. White Contempt = true in all case. Here are the result against Komodo Contempt = 0 of Contempt 25,50,75,100.
It is pretty sure Contempt = 50 performs significantly better than Contempt = 75 or all other contempts
Overall, White performance is 47.1%.
But Contempt = 75 against Contempt = 0 shows a White win performance at 44.5%, as in earlier tests. While the more adequate Contempt = 50 shows a White win performance of 50.2%. The difference in White performance in these two cases is significant. So, it might be that the low White win ratio of Contempt = 75 is partly due to an inadequately chosen value of the Contempt (aside identical Komodo engines in earlier tests)
Will check now self-play of Contempt = 50 Komodos.
I ran the following test last night. Komodo 13.2.5 (regular mode) vs K2453.00 MCTS (latest dev), with regular K on one thread and MCTS on2, to make them realsonably competitive (regular won by 39 elo). 2 min + 1", White Contempt 75 (maybe I need to rerun it with Contempt 50 in view of your results), all 358 positions from your NBSC book with both colors so 716 games. Result: White won 351, Draws 326, Black won 36, so 49.2% White wins. I know this adds up to just 713 games, I don't know why the other three are missing from the results file. Anyway a 49.2% for White is just about what I would call optimum. All in all, very promising.
Laskos wrote: ↑Mon Dec 16, 2019 6:56 am
Second, Lc0 testing, between best t40 and t30 nets:
2000 games at 100 nodes/move:
+1064 =386 -550 62.8%
White score: 53.2%
1000 games at 1000 nodes/move
+514 =303 -183 66.5%
White score: 51.4%
Draw rate increases to longer TC, Armageddon score decreases from above towards 50%. Again, a promising result.
And again good result at 5000 nodes/move Lc0 testing between best t40 and t30 nets:
1000 games at 5000 nodes/move
+509 =366 -125 69.2%
White score: 50.9%
Nice progression from lower nodes count. AB engines have 48.3% White score at similar strength. All in all, very balanced and seems to slowly converge towards close to 50%.
I decided to check out your one bad result, Komodo self-play, to see if it got better or worse with more time and more threads. At 2' + 1" on four threads, latest Komodo dev. got 79 wins, 98 draws, and one loss for 44.4% Armageddon score, somewhat better than your result for selfplay. So at least it's moving in the right direction. With your results for non-selfplay averaging around 49.6% (using only the longest TC for Lc0), looks like this version is most likely as close as we can hope to get to being fair without crossing the line of White having a forced win (it seems). It's better to be below this line than above it, so I think this may be the perfect Armageddon variant at last! I'll try to get some more data and look for ways to promote the idea.
Some experiments with Komodo's Contempt. White Contempt = true in all case. Here are the result against Komodo Contempt = 0 of Contempt 25,50,75,100.
It is pretty sure Contempt = 50 performs significantly better than Contempt = 75 or all other contempts
Overall, White performance is 47.1%.
But Contempt = 75 against Contempt = 0 shows a White win performance at 44.5%, as in earlier tests. While the more adequate Contempt = 50 shows a White win performance of 50.2%. The difference in White performance in these two cases is significant. So, it might be that the low White win ratio of Contempt = 75 is partly due to an inadequately chosen value of the Contempt (aside identical Komodo engines in earlier tests)
Will check now self-play of Contempt = 50 Komodos.
I ran the following test last night. Komodo 13.2.5 (regular mode) vs K2453.00 MCTS (latest dev), with regular K on one thread and MCTS on2, to make them realsonably competitive (regular won by 39 elo). 2 min + 1", White Contempt 75 (maybe I need to rerun it with Contempt 50 in view of your results), all 358 positions from your NBSC book with both colors so 716 games. Result: White won 351, Draws 326, Black won 36, so 49.2% White wins. I know this adds up to just 713 games, I don't know why the other three are missing from the results file. Anyway a 49.2% for White is just about what I would call optimum. All in all, very promising.
So I reran the same test with White Contempt = 50 instead of 75 per your suggestion, and the White result did improve, climbing from 49.2% to 49.9% (!), 356 White wins, 332 draws, 26 Black wins, 2 games missing. If just one of the draws had been a White win it would have been a total tie with Armageddon scoring. Note that the main point of this test is that it allows for two quite different engines (Komodo and Komodo MCTS) to play while both still are able to use White Contempt. I guess this was pretty much the only way to accomplish this.
lkaufman wrote: ↑Tue Dec 17, 2019 6:22 pm
I ran the following test last night. Komodo 13.2.5 (regular mode) vs K2453.00 MCTS (latest dev), with regular K on one thread and MCTS on2, to make them realsonably competitive (regular won by 39 elo). 2 min + 1", White Contempt 75 (maybe I need to rerun it with Contempt 50 in view of your results), all 358 positions from your NBSC book with both colors so 716 games. Result: White won 351, Draws 326, Black won 36, so 49.2% White wins. I know this adds up to just 713 games, I don't know why the other three are missing from the results file. Anyway a 49.2% for White is just about what I would call optimum. All in all, very promising.
So I reran the same test with White Contempt = 50 instead of 75 per your suggestion, and the White result did improve, climbing from 49.2% to 49.9% (!), 356 White wins, 332 draws, 26 Black wins, 2 games missing. If just one of the draws had been a White win it would have been a total tie with Armageddon scoring. Note that the main point of this test is that it allows for two quite different engines (Komodo and Komodo MCTS) to play while both still are able to use White Contempt. I guess this was pretty much the only way to accomplish this.
Yes, that's pretty much the only way to use Colored Contempt with different engines (and the AB and MCTS versions are quite different in Sim test). So, on a wide range of time controls (but not higher than Blitz), engines show a very balanced White/Black Armageddon score. This 2min + 1s should be already a superhuman level playing, and the issue is not strength, but the "style and mood, psychology" of humans. I don't think that the opening theory of this variant will change dramatically the balance. After all, patzer in openings engines of say 2002 were giving similar White/Black balance in regular Chess to that of strong humans who knew the opening theory well. But you must have a much more evolved image of what would strong humans do in NBSC, I can only speculate.
I combined the 5-mover opening suites built with Komodo on 4 threads earlier with Contempt = 75 and now with Contempt = 50 for a total of 519 different positions. Here is the link, as far as to my patzer eyes, the openings seem reasonable:
lkaufman wrote: ↑Tue Dec 17, 2019 6:22 pm
I ran the following test last night. Komodo 13.2.5 (regular mode) vs K2453.00 MCTS (latest dev), with regular K on one thread and MCTS on2, to make them realsonably competitive (regular won by 39 elo). 2 min + 1", White Contempt 75 (maybe I need to rerun it with Contempt 50 in view of your results), all 358 positions from your NBSC book with both colors so 716 games. Result: White won 351, Draws 326, Black won 36, so 49.2% White wins. I know this adds up to just 713 games, I don't know why the other three are missing from the results file. Anyway a 49.2% for White is just about what I would call optimum. All in all, very promising.
So I reran the same test with White Contempt = 50 instead of 75 per your suggestion, and the White result did improve, climbing from 49.2% to 49.9% (!), 356 White wins, 332 draws, 26 Black wins, 2 games missing. If just one of the draws had been a White win it would have been a total tie with Armageddon scoring. Note that the main point of this test is that it allows for two quite different engines (Komodo and Komodo MCTS) to play while both still are able to use White Contempt. I guess this was pretty much the only way to accomplish this.
Yes, that's pretty much the only way to use Colored Contempt with different engines (and the AB and MCTS versions are quite different in Sim test). So, on a wide range of time controls (but not higher than Blitz), engines show a very balanced White/Black Armageddon score. This 2min + 1s should be already a superhuman level playing, and the issue is not strength, but the "style and mood, psychology" of humans. I don't think that the opening theory of this variant will change dramatically the balance. After all, patzer in openings engines of say 2002 were giving similar White/Black balance in regular Chess to that of strong humans who knew the opening theory well. But you must have a much more evolved image of what would strong humans do in NBSC, I can only speculate.
I combined the 5-mover opening suites built with Komodo on 4 threads earlier with Contempt = 75 and now with Contempt = 50 for a total of 519 different positions. Here is the link, as far as to my patzer eyes, the openings seem reasonable:
They seem to diverge from regular opening theory often pretty substantially.
I studied your earlier 358 position book a bit. One issue was that some openings were repeated with only tiny changes, maybe the final developing move was changed, but transposition was likely. This doesn't invalidate anything, it just means that sample error is a bit larger than what pure math would say since the games aren't totally independent. I also noticed some clearly inferior moves in some lines, like rather pointless h3 or h6 moves, but I think that applied to both colors so probably no real bias involved. The best White first moves are pretty much the same as in normal chess, but quickly things diverge. For example the Albin Counter Gambit (1.d4 d5 2.c4 e5!?) is probably a good defense in NBSC because it aims at queenside castling, and so White might even do better to avoid it by 2.Nf3 or by 1.c4 or 1.e4. It will be very difficult to prove what opening is best for each side, just as it is in normal chess. It's harder for chessplayers (human or engine) to decide which of two dubious positions is less bad than it is just to say which side stands better.
lkaufman wrote: ↑Tue Dec 17, 2019 6:22 pm
I ran the following test last night. Komodo 13.2.5 (regular mode) vs K2453.00 MCTS (latest dev), with regular K on one thread and MCTS on2, to make them realsonably competitive (regular won by 39 elo). 2 min + 1", White Contempt 75 (maybe I need to rerun it with Contempt 50 in view of your results), all 358 positions from your NBSC book with both colors so 716 games. Result: White won 351, Draws 326, Black won 36, so 49.2% White wins. I know this adds up to just 713 games, I don't know why the other three are missing from the results file. Anyway a 49.2% for White is just about what I would call optimum. All in all, very promising.
So I reran the same test with White Contempt = 50 instead of 75 per your suggestion, and the White result did improve, climbing from 49.2% to 49.9% (!), 356 White wins, 332 draws, 26 Black wins, 2 games missing. If just one of the draws had been a White win it would have been a total tie with Armageddon scoring. Note that the main point of this test is that it allows for two quite different engines (Komodo and Komodo MCTS) to play while both still are able to use White Contempt. I guess this was pretty much the only way to accomplish this.
Yes, that's pretty much the only way to use Colored Contempt with different engines (and the AB and MCTS versions are quite different in Sim test). So, on a wide range of time controls (but not higher than Blitz), engines show a very balanced White/Black Armageddon score. This 2min + 1s should be already a superhuman level playing, and the issue is not strength, but the "style and mood, psychology" of humans. I don't think that the opening theory of this variant will change dramatically the balance. After all, patzer in openings engines of say 2002 were giving similar White/Black balance in regular Chess to that of strong humans who knew the opening theory well. But you must have a much more evolved image of what would strong humans do in NBSC, I can only speculate.
I combined the 5-mover opening suites built with Komodo on 4 threads earlier with Contempt = 75 and now with Contempt = 50 for a total of 519 different positions. Here is the link, as far as to my patzer eyes, the openings seem reasonable:
They seem to diverge from regular opening theory often pretty substantially.
Using your new larger book, I reran my Komodo AB vs MCTS match with Contempt 50 but at a faster TC (30" + .3"), so 1033 games (five didn't show up in the file). This time White won 523, drew 443, Black won 67, so White won 50.6%. Yet again remarkably close to 50% and not measurably different than the 49.9% at 2' + 1".
lkaufman wrote: ↑Tue Dec 17, 2019 6:22 pm
I ran the following test last night. Komodo 13.2.5 (regular mode) vs K2453.00 MCTS (latest dev), with regular K on one thread and MCTS on2, to make them realsonably competitive (regular won by 39 elo). 2 min + 1", White Contempt 75 (maybe I need to rerun it with Contempt 50 in view of your results), all 358 positions from your NBSC book with both colors so 716 games. Result: White won 351, Draws 326, Black won 36, so 49.2% White wins. I know this adds up to just 713 games, I don't know why the other three are missing from the results file. Anyway a 49.2% for White is just about what I would call optimum. All in all, very promising.
So I reran the same test with White Contempt = 50 instead of 75 per your suggestion, and the White result did improve, climbing from 49.2% to 49.9% (!), 356 White wins, 332 draws, 26 Black wins, 2 games missing. If just one of the draws had been a White win it would have been a total tie with Armageddon scoring. Note that the main point of this test is that it allows for two quite different engines (Komodo and Komodo MCTS) to play while both still are able to use White Contempt. I guess this was pretty much the only way to accomplish this.
Yes, that's pretty much the only way to use Colored Contempt with different engines (and the AB and MCTS versions are quite different in Sim test). So, on a wide range of time controls (but not higher than Blitz), engines show a very balanced White/Black Armageddon score. This 2min + 1s should be already a superhuman level playing, and the issue is not strength, but the "style and mood, psychology" of humans. I don't think that the opening theory of this variant will change dramatically the balance. After all, patzer in openings engines of say 2002 were giving similar White/Black balance in regular Chess to that of strong humans who knew the opening theory well. But you must have a much more evolved image of what would strong humans do in NBSC, I can only speculate.
I combined the 5-mover opening suites built with Komodo on 4 threads earlier with Contempt = 75 and now with Contempt = 50 for a total of 519 different positions. Here is the link, as far as to my patzer eyes, the openings seem reasonable:
They seem to diverge from regular opening theory often pretty substantially.
Using your new larger book, I reran my Komodo AB vs MCTS match with Contempt 50 but at a faster TC (30" + .3"), so 1033 games (five didn't show up in the file). This time White won 523, drew 443, Black won 67, so White won 50.6%. Yet again remarkably close to 50% and not measurably different than the 49.9% at 2' + 1".
With Komodo MCTS at 30 + 0.3s and Komodo AB at 20 + 0.2s, both White Contempt = 50, in 800 games from the larger opening suite:
White scoring:
+394 =350 -56
49.3% Armageddon scoring.
All in all, in thousands of games somethings close to 49.5% or so, average with different engines, time controls, AB and MCTS. Not diverging anyway with time control. I am experimenting now with sensitiveness to strength (resolving power), it seems some 1.25-1.30x higher for NBSC compared to regular Chess, meaning that it would need 1.6x factor less games to the same conclusiveness. But this is side and reversed. Individual games with Armageddon scoring seem to exhibit only some 1.15-1.20x higher sensitivity, but individual games also exhibit White/Black asymmetry, which is hard to guess for me how it is felt by strong humans.
I will try to build now a reasonable book (EPD suite) with Lc0. It doesn't have Contempt, though.
I will try to build a larger book by widening a bit the trapezoid at its narrow end-side, and by allowing an offset of 3% win probability instead of 2%.
I will try to build a larger book by widening a bit the trapezoid at its narrow end-side, and by allowing an offset of 3% win probability instead of 2%.
I have built a larger 6-mover opening suite with Lc0 with 476 positions: https://file.io/w77hc1
But I think the openings in it are discrediting Black's chances. Lc0 has an "in innate" Contempt for the side to move (I have shown that in the past), which in openings means that White plays well, while Black is wrongly aggressive, instead of going to resistance for a draw. Look at the openings, aren't many of them exhibiting that? As expected the White performance goes up with wrong for Black Lc0 openings:
1000 games at 30 + 0.3s
SF_9 vs K_131 (White Contempt = 50)
52.5%
1000 games
K_AB (20+0.2s) vs K_MCTS (30+0.3s) both White Contempt = 50
55.1%
1000 games, 1000 nodes/move
Lc0 T40 vs Lc0 T30
52.1%
==================
53.2% White performance (Armageddon scoring) in 3000 games with openings built using Lc0.
I think the only reliable way to build representative openings for this unbalanced variant is by using Komodo with its Colored Contempt. Contempt usage IS important from the start of the game.
