87% of the poll voted that SF will win Tcec Sufi 19.

[Guenther]

Indeed not sure the point of arguing with you when you deny obvious facts admitted by TCEC themselves.
Note, no one except you is arguing that the book is not unbalanced, but whether it is fair or not...

...

What TCEC has 'written' is still the same text as for several seasons ago already, as usual they are too lazy in changing info text.
Surely the openings were less balanced in the past and the text originally was written for another season.

And another important thing, which Jonathan already mentioned, why should we trust SF evals in the first place here?
Shouldn't LC0 be much better in evaluating opening positions, that's what most people say, so why not show its evals out of book too ;-)

I can agree that 'unbalanced' positions occur (only talking about this season!), but you have listed some between 0.66- 0.80 SF eval
and even most above 1.0 SF eval are no deadends. The unbalanced part so far is quite a tiny portion IMHO, but I have no beef here
and no interest in arguing this overly.

[mwyoung]

I am surprized you need clarification for this but here it goes.
These are scores of stockfish right out of book for the second half of the match starting from game 52

52 - +1.31
54 - +1.06
56 - +1.19
58 - +0.80
60 - +1.14
62 - +1.17
64 - +1.17
66 - +1.12
68 - +1.20
70 - +0.80
72 - -0.93
74 - +0.70
76 - +0.66
78 - +1.38
80 - +1.05
82 - +0.83

SF may well be superior to Lc0 but this unbalanced openings prove nothing.
Even SF without NNUE was able to get +7 over leela last season entirely due to the book.

Unless Stockfish has contempt = 0 and analysis contempt turned off, Stockfish evals are not a reliable indicator of the true nature of the opening.

What point are you trying to argue here. That TCEC are liars.


From TCEC------"Q: To what degree do you bias openings?


A: In League 1 Nelson is in his comfort zone when opening evaluations in traditional chess engines (i.e. not neural nets) are between +/-0.40 and +/-0.65. In Premier League, a score of +/-0.50 to +/-0.80 is more common. Sometimes he will deviate from these guidelines to the upside or downside to satisfying opening variety goals.

In each season’s Superfinal Jeroen has a free hand to do whatever he thinks will result in an interesting and varied contest. Sometimes his book-exit evaluations will exceed +/-1.00; sometimes he will offer speculative gambits. His goal is to keep the draw-rate in the 65-80% range each season without a surfeit of one-sided openings."

[Dann Corbit]

Second half, 5 mini-match wins

52 - +1.31 + -
54 - +1.06 + = won
56 - +1.19 + -
58 - +0.80 =
60 - +1.14 + = won
62 - +1.17 =
64 - +1.17 =
66 - +1.12 =
68 - +1.20 + -
70 - +0.80 =
72 - -0.93 =
74 - +0.70 + = won
76 - +0.66 + = won
78 - +1.38 + -
80 - +1.05 + = won
82 - +0.83 =

won=SF wins minimatch

Something interesting here is that the most polar positions were not wins

Code: Select all

+0.66 + =  won
+0.70 + =  won
+0.80 =
+0.80 =
+0.83 =
+1.05 + =  won
+1.06 + =  won
+1.12 =
+1.14 + =  won
+1.17 =
+1.17 =
+1.19 + -
+1.20 + -
+1.31 + -
+1.38 + -
-0.93 =

None of the positions with an offset of 1.17 or more was a win for SF.
I think Kai also had an interesting argument. LC0 is trained for quiet positions. If they added training for lopsided positions, perhaps LC0 would fare better.
I guess it is kind of an expected outcome, now that I think of it. If we do a mind experiment where one side falls out of book up a queen, we will see +- for all of those matches. Indeed, that is what we saw with every opening with a score above +117 centipawns.
So it looks to me like the danger zone for LC0 is capped right around + 115 centipawns.
Not enough games to decide this of course, just a mind experiment and speculation on my part.

[mwyoung]

Second half, 5 mini-match wins

52 - +1.31 + -
54 - +1.06 + = won
56 - +1.19 + -
58 - +0.80 =
60 - +1.14 + = won
62 - +1.17 =
64 - +1.17 =
66 - +1.12 =
68 - +1.20 + -
70 - +0.80 =
72 - -0.93 =
74 - +0.70 + = won
76 - +0.66 + = won
78 - +1.38 + -
80 - +1.05 + = won
82 - +0.83 =

won=SF wins minimatch

Something interesting here is that the most polar positions were not wins

Code: Select all

+0.66 + =  won
+0.70 + =  won
+0.80 =
+0.80 =
+0.83 =
+1.05 + =  won
+1.06 + =  won
+1.12 =
+1.14 + =  won
+1.17 =
+1.17 =
+1.19 + -
+1.20 + -
+1.31 + -
+1.38 + -
-0.93 =

None of the positions with an offset of 1.17 or more was a win for SF.
I think Kai also had an interesting argument. LC0 is trained for quiet positions. If they added training for lopsided positions, perhaps LC0 would fare better.
I guess it is kind of an expected outcome, now that I think of it. If we do a mind experiment where one side falls out of book up a queen, we will see +- for all of those matches. Indeed, that is what we saw with every opening with a score above +117 centipawns.
So it looks to me like the danger zone for LC0 is capped right around + 115 centipawns.
Not enough games to decide this of course, just a mind experiment and speculation on my part.

Chess stats is a game of odds. And I know SF, and Lc0 better then most. So I was one of the 87% under these playing conditions. But this is not true chess, or testing. But a match of who can play garbage openings better.

"87% of the poll voted that SF will win Tcec Sufi 19."

[mwyoung]

Indeed not sure the point of arguing with you when you deny obvious facts admitted by TCEC themselves.
Note, no one except you is arguing that the book is not unbalanced, but whether it is fair or not...

...

What TCEC has 'written' is still the same text as for several seasons ago already, as usual they are too lazy in changing info text.
Surely the openings were less balanced in the past and the text originally was written for another season.

And another important thing, which Jonathan already mentioned, why should we trust SF evals in the first place here?
Shouldn't LC0 be much better in evaluating opening positions, that's what most people say, so why not show its evals out of book too

I can agree that 'unbalanced' positions occur (only talking about this season!), but you have listed some between 0.66- 0.80 SF eval
and even most above 1.0 SF eval are no deadends. The unbalanced part so far is quite a tiny portion IMHO, but I have no beef here
and no interest in arguing this overly.

"and even most above 1.0 SF eval are no deadends."

I would not want to argue this point either. As many are "deadends".

[OneTrickPony]

Chess stats is a game of odds. And I know SF, and Lc0 better then most. So I was one of the 87% under these playing conditions. But this is not true chess, or testing. But a match of who can play garbage openings better.

It seems to me that those engines are so strong that we can test one of the two:

1)"Garbage openings" (that is positions with significant edge for one side) at long time controls/big hardware
2)Normal openings at very fast time controls/weaker hardware

Otherwise we would just get 100% or close to it draws and it would be boring as hell.

I think one interesting question is: "Is there ever going to be an engine that beats current SF in main line openings on very strong hardware (say TCEC hardware). I would love to see a poll on that.

Btw I think testing "garbage openings" is very valuable. Engines are mainly used for analysis/learning and the ones that quickly point out mistakes and are able to show correct plan refuting them are very valuable for human players. It's also more fun to watch if we start from unbalanced positions.

I love current TCEC book: some normal(ish) positions in case one of the engines is strong enough to show the edge there (it didn't happen) and a lot of marginal stuff where smaller advantage in playing strength is enough to win the pair of games.

[Daniel Shawul]

Second half, 5 mini-match wins

52 - +1.31 + -
54 - +1.06 + = won
56 - +1.19 + -
58 - +0.80 =
60 - +1.14 + = won
62 - +1.17 =
64 - +1.17 =
66 - +1.12 =
68 - +1.20 + -
70 - +0.80 =
72 - -0.93 =
74 - +0.70 + = won
76 - +0.66 + = won
78 - +1.38 + -
80 - +1.05 + = won
82 - +0.83 =

won=SF wins minimatch

Something interesting here is that the most polar positions were not wins

Code: Select all

+0.66 + =  won
+0.70 + =  won
+0.80 =
+0.80 =
+0.83 =
+1.05 + =  won
+1.06 + =  won
+1.12 =
+1.14 + =  won
+1.17 =
+1.17 =
+1.19 + -
+1.20 + -
+1.31 + -
+1.38 + -
-0.93 =

None of the positions with an offset of 1.17 or more was a win for SF.
I think Kai also had an interesting argument. LC0 is trained for quiet positions. If they added training for lopsided positions, perhaps LC0 would fare better.
I guess it is kind of an expected outcome, now that I think of it. If we do a mind experiment where one side falls out of book up a queen, we will see +- for all of those matches. Indeed, that is what we saw with every opening with a score above +117 centipawns.
So it looks to me like the danger zone for LC0 is capped right around + 115 centipawns.
Not enough games to decide this of course, just a mind experiment and speculation on my part.

In training, the balance between exploring different type of positions and learning the best lines well is controlled by the temperature parameter.
If an 800 node search on the start position does not visit a move well enough, it won't be trained up on much.
For this, g4 is hardly trained upon compared to e4 for example, and I am pretty sure an AB engine will do well in these circumstances.
There is also the network size to consider because you can only learn so much, so you have to choose what you want to learn well.

I think the best solution is to do training with a book, where instead of letting the engine figure out the openings, a book (e.g. TCEC's book) is forced as the opening. This has been tried already and it was found that it can accelerate learning initially but I think it didn't make a difference to overall strength eventually. If TCEC is the goal, a network can be trained specificallly for unbalanced openings.

Note that the high scores >= 1.19 are won by both sides, so like you said there is probably score below that in which SF wins with white and draws with black. I think if openings were chosen by lc0 to be >1.0, both sides will just win their games equally since lc0 squares often understiate the value of the position.

[mwyoung]

Chess stats is a game of odds. And I know SF, and Lc0 better then most. So I was one of the 87% under these playing conditions. But this is not true chess, or testing. But a match of who can play garbage openings better.

It seems to me that those engines are so strong that we can test one of the two:

1)"Garbage openings" (that is positions with significant edge for one side) at long time controls/big hardware
2)Normal openings at very fast time controls/weaker hardware

Otherwise we would just get 100% or close to it draws and it would be boring as hell.

I think one interesting question is: "Is there ever going to be an engine that beats current SF in main line openings on very strong hardware (say TCEC hardware). I would love to see a poll on that.

Btw I think testing "garbage openings" is very valuable. Engines are mainly used for analysis/learning and the ones that quickly point out mistakes and are able to show correct plan refuting them are very valuable for human players. It's also more fun to watch if we start from unbalanced positions.

I love current TCEC book: some normal(ish) positions in case one of the engines is strong enough to show the edge there (it didn't happen) and a lot of marginal stuff where smaller advantage in playing strength is enough to win the pair of games.

"Otherwise we would just get 100% or close to it draws and it would be boring as hell."

Yes, you will get close to that score, because chess with equal and strong players. Theory says you will expect that score. Regardless of time controls and hardware. And we also see this from the best human players in the world. Lots of draws, unless they play a chess engine. Hmmm.

And I strongly disagree that draws are boring. Great chess is great chess for the two best chess engines in the world. And when you do see a win under fair conditions. It is pure magic!

[peter]

People talked too soon in this thread. SF is up by +6 points now on TCEC.

After 81 games:

Wins = 14
Draws = 59
Losses = 8

In the meantime at game 84 SF is winning another one full point with remis in the same opening in game 83.
10- moves Sicilian, SF- eval at move 11 was 91 cp, LC0- eval 36 cp.

1. e4 c5 2. Nf3 Nc6 3. d4 cxd4 4. Nxd4 Qc7 5. Nc3 e6 6. Be3 a6 7. a3 Nf6 8. f4 d6 9. Qf3 Be7 10. O-O-O O-O

After 30 moves SF- eval is 3.91, LC0's 2.72.

I like the discussion about engine- evals to judge opening- positions, and I like Jeroen Noomen's test positions for TCEC, but I admit, it's a matter of taste, what you like to see in chess and what you don't like.

[Milos]

I do not want TCEC to change any rules. But some people need to understand the RULES.

Garbage openings, garbage results.

Testing conditions matter!

Garbage openings only for garbage engines. And enormous pain for fanboys of a given garbage engine .