We have been testing computer engines the wrong way.......

[Chessqueen]

I truly believe that computer engines different than humans that have to memorize long openings and endgames, do NOT have to have an opening with more than 5 moves at the most because we are taking away their capabilities of using their algorithms and calculating power to create new openings for us, and the same goes for endgames that are more than 4 men TB. In another word computer engines are only being tested to see which engines are stronger in the middle game. My personal observation is that if we want to really know which engines are the best we should limit the opening to no more than 5 moves. In my observation Stockfish won the TCEC by 50.5 to 49.5 and now LCZero is showing Stockfish that without opening LCZero understand how to open up better than Stockfish. Anyway there should be a middle ground to avoid repetitions of same openings over and over again and that could be accomplish by playing games with 5 moves opening and let the engines be creative from that point on. Let me repeat, I believe that all that we are testing is which engines are better at the middlegame stage.
https://tcec.chessdom.com/

PS: Of course some people would say well that is why we have Chess960, but some openings of Chess960 are more favorable to one side, unless each opening is reverse where on engine play White and then the Black side of the same Chess960 position

[Michael Sherwin]

I agree mostly. However, if an engine does better playing 1. d4 than when it plays 1. e4 and it is forced to play 1. e4 then the true elo of the engine is not being tested.

[Chessqueen]

I agree mostly. However, if an engine does better playing 1. d4 than when it plays 1. e4 and it is forced to play 1. e4 then the true elo of the engine is not being tested.

Correct , I see your point, but all these openings that we are using to test engines can still be used, but they have to be limited to only 5 moves and let the engine be creative from that point on, since most of the openings that we use to test engines were created by humans, and computer engines are far better than humans were or have ever being, therefore why follow X lines and so on. and the same goes for using EGT of more than 5 men TB, simply let the engines figure out what is the best continuations and we can judge which engines are best for openings, middle games and endgames, NOT just Middle Game as we are testing at the moment.
https://tcec.chessdom.com/

[hgm]

What is the right way to test all depends on what property you want to test. If you want to know how good on average, or how reliable an engine is for analyzing an arbitrary Chess position, its ability to win games from the starting position can be completely irrelevant. (E.g. look at LeelaChess Zero, which is extremely adept at winning games, yet can make very simple tactical blunders. In games it knows how to avoid positions where it would make such blunders, but that doesn't make it suitable for general analysis. Also, being good on average doesn't imply reliability, which means good in the worst case.)

So indeed, measuring Elo by playing games really only measures how good the engines are on average at winning games from the set of book positions, against other, very similar engines. As most engine users are not interested in playing engine-engine games, this isn't really a quantity of interest.

[jdart]

I believe most engines still play quite poorly in the opening. I regularly see them play into known inferior lines and then lose.

--Jon

[Chessqueen]

What is the right way to test all depends on what property you want to test. If you want to know how good on average, or how reliable an engine is for analyzing an arbitrary Chess position, its ability to win games from the starting position can be completely irrelevant. (E.g. look at LeelaChess Zero, which is extremely adept at winning games, yet can make very simple tactical blunders. In games it knows how to avoid positions where it would make such blunders, but that doesn't make it suitable for general analysis. Also, being good on average doesn't imply reliability, which means good in the worst case.)

So indeed, measuring Elo by playing games really only measures how good the engines are on average at winning games from the set of book positions, against other, very similar engines. As most engine users are not interested in playing engine-engine games, this isn't really a quantity of interest.

You are correct, after I spent several days watching LCZero Vs Stockfish playing at TCEC, I observed that the same Opening were chosen by both engines, spcecialy LCZero who is winning the Non Book Match, a lot of the same opening kept being repeated until move 12 were a deviation or a different move is chosen by Stockfisk, but SF does NOT have an algorithm that prevent it from playing the same Nadolf , Sicilian or Berlin. To solve this if TCEC is willing to continue every year testing engines without Book Opening, at least they can select 50 Openings limited to 4 moves, therefore, other openings such as the English, Danish, Scotch, Caro Kann, Slav Def, Queen Gambit, Nimzo Indian Def, and many many other Openings, but limited to only 4 or 5 moves. I also believe that if the EGT TB is removed Stockfish can play the endgame better than LCZero, even if the best or strongest NN Net is used such as 32930 with 20.2 cpu 3.4 and cpuctbase = 10K for Leela.