Rebel wrote: ↑Sat Aug 22, 2026 7:00 pm
EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
In case of high draw rate and 49.5/50.5 result and 50,000 games, assuming you’re testing a base against a trial mod, best just assume the trial mod was not worth the trouble and stay with base, or?
Rebel wrote: ↑Sat Aug 22, 2026 7:00 pm
EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
In case of high draw rate and 49.5/50.5 result and 50,000 games, assuming you’re testing a base against a trial mod, best just assume the trial mod was not worth the trouble and stay with base, or?
The thing I dislike about SPRT for is that it aborts a match based on an algorithm dealing with randomness. I like to decide that myself. If one has knowledge how error bars works one does not need SPRT.
Rebel wrote: ↑Sat Aug 22, 2026 7:00 pm
EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
In case of high draw rate and 49.5/50.5 result and 50,000 games, assuming you’re testing a base against a trial mod, best just assume the trial mod was not worth the trouble and stay with base, or?
The thing I dislike about SPRT for is that it aborts a match based on an algorithm dealing with randomness. I like to decide that myself. If one has knowledge how error bars works one does not need SPRT.
Rebel wrote: ↑Sat Aug 22, 2026 7:00 pm
EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
In case of high draw rate and 49.5/50.5 result and 50,000 games, assuming you’re testing a base against a trial mod, best just assume the trial mod was not worth the trouble and stay with base, or?
The thing I dislike about SPRT for is that it aborts a match based on an algorithm dealing with randomness. I like to decide that myself. If one has knowledge how error bars works one does not need SPRT.
Look at the number of games played by an engine and watch the corresponding error bar.
I tried the Stockfish SPRT calculator, much in sync with my tool.
Why is SF so successful?
Because they have the hardware to measure 1 elo.
I do not like the explanation that SF is succesful because they have the hardware to measure 1 elo.
This is not explanation that we can learn something about chess about it.
Stockfish is succesful because it know a lot more about chess relative to other engines and it is more interesting to define what is the knowledge that stokcifsh has and other do not have.
For example the way to try to do it for knowledge in the evaluation function is to analyze a lot of games with stockfish and other engines and to see positions when stockfish disagree with other engines even at small depths.
After you do it try to define for every case what stockfish knows that the second engine does not know when the idea is to generalize so this definition is relevant not only for one position.
Rebel wrote: ↑Sat Aug 22, 2026 7:00 pm
EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
In case of high draw rate and 49.5/50.5 result and 50,000 games, assuming you’re testing a base against a trial mod, best just assume the trial mod was not worth the trouble and stay with base, or?
The thing I dislike about SPRT for is that it aborts a match based on an algorithm dealing with randomness. I like to decide that myself. If one has knowledge how error bars works one does not need SPRT.
Look at the number of games played by an engine and watch the corresponding error bar.
I tried the Stockfish SPRT calculator, much in sync with my tool.
Why is SF so successful?
Because they have the hardware to measure 1 elo.
I do not like the explanation that SF is succesful because they have the hardware to measure 1 elo.
This is not explanation that we can learn something about chess about it.
Stockfish is succesful because it know a lot more about chess relative to other engines and it is more interesting to define what is the knowledge that stokcifsh has and other do not have.
For example the way to try to do it for knowledge in the evaluation function is to analyze a lot of games with stockfish and other engines and to see positions when stockfish disagree with other engines even at small depths.
After you do it try to define for every case what stockfish knows that the second engine does not know when the idea is to generalize so this definition is relevant not only for one position.
Novelty has always been the main reason for progress, but not without proper scientific testing. In the 80's and 90's by the lack of good hardware programmers were limited playing 100-500 games max. We do know now how unreliable that was. Changes that looked good based on too few games in reality could have been lowering elo, changes we thought were regressions (and thus rejected) in reality could have been improvements. I have no doubt both cases have happened to me frequently.
Christophe Theron was the first one who understood, he had 2 poor Pentium 90, he could play 800 games twice a day with the focus on search ideas and it put him on top for a couple of years.
Vas came and understood also, he invested in hardware playing thousands of games to test a change, put him on top for 5-6 years. Vas also said, re-check your old failed ideas, they might be improvements after all.
90% of coding is debugging, the other 10% is writing bugs.
Rebel wrote: ↑Sat Aug 22, 2026 7:00 pm
EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
In case of high draw rate and 49.5/50.5 result and 50,000 games, assuming you’re testing a base against a trial mod, best just assume the trial mod was not worth the trouble and stay with base, or?
The thing I dislike about SPRT for is that it aborts a match based on an algorithm dealing with randomness. I like to decide that myself. If one has knowledge how error bars works one does not need SPRT.
Look at the number of games played by an engine and watch the corresponding error bar.
I tried the Stockfish SPRT calculator, much in sync with my tool.
Why is SF so successful?
Because they have the hardware to measure 1 elo.
I do not like the explanation that SF is succesful because they have the hardware to measure 1 elo.
This is not explanation that we can learn something about chess about it.
Stockfish is succesful because it know a lot more about chess relative to other engines and it is more interesting to define what is the knowledge that stokcifsh has and other do not have.
For example the way to try to do it for knowledge in the evaluation function is to analyze a lot of games with stockfish and other engines and to see positions when stockfish disagree with other engines even at small depths.
After you do it try to define for every case what stockfish knows that the second engine does not know when the idea is to generalize so this definition is relevant not only for one position.
Novelty has always been the main reason for progress, but not without proper scientific testing. In the 80's and 90's by the lack of good hardware programmers were limited playing 100-500 games max. We do know now how unreliable that was. Changes that looked good based on too few games in reality could have been lowering elo, changes we thought were regressions (and thus rejected) in reality could have been improvements. I have no doubt both cases have happened to me frequently.
Christophe Theron was the first one who understood, he had 2 poor Pentium 90, he could play 800 games twice a day with the focus on search ideas and it put him on top for a couple of years.
Vas came and understood also, he invested in hardware playing thousands of games to test a change, put him on top for 5-6 years. Vas also said, re-check your old failed ideas, they might be improvements after all.
Of course testing is the factor that made the top engines strong but for our knowledge about chess we need an alternative explanation because I know nothing more about chess from the explanation of testing when you know more about chess for the explanation about a better way to evaluate chess position. Humans who play chess who want to know why stockfish played a move in many cases so they can also make better moves certainaly need better explanation.
Rebel wrote: ↑Sat Aug 22, 2026 7:00 pm
EMULATE is a tool that creates random matches between 2 identical hypothetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.
In case of high draw rate and 49.5/50.5 result and 50,000 games, assuming you’re testing a base against a trial mod, best just assume the trial mod was not worth the trouble and stay with base, or?
The thing I dislike about SPRT for is that it aborts a match based on an algorithm dealing with randomness. I like to decide that myself. If one has knowledge how error bars works one does not need SPRT.
Look at the number of games played by an engine and watch the corresponding error bar.
I tried the Stockfish SPRT calculator, much in sync with my tool.
Why is SF so successful?
Because they have the hardware to measure 1 elo.
I do not like the explanation that SF is succesful because they have the hardware to measure 1 elo.
This is not explanation that we can learn something about chess about it.
Stockfish is succesful because it know a lot more about chess relative to other engines and it is more interesting to define what is the knowledge that stokcifsh has and other do not have.
For example the way to try to do it for knowledge in the evaluation function is to analyze a lot of games with stockfish and other engines and to see positions when stockfish disagree with other engines even at small depths.
After you do it try to define for every case what stockfish knows that the second engine does not know when the idea is to generalize so this definition is relevant not only for one position.
Novelty has always been the main reason for progress, but not without proper scientific testing. In the 80's and 90's by the lack of good hardware programmers were limited playing 100-500 games max. We do know now how unreliable that was. Changes that looked good based on too few games in reality could have been lowering elo, changes we thought were regressions (and thus rejected) in reality could have been improvements. I have no doubt both cases have happened to me frequently.
Christophe Theron was the first one who understood, he had 2 poor Pentium 90, he could play 800 games twice a day with the focus on search ideas and it put him on top for a couple of years.
Vas came and understood also, he invested in hardware playing thousands of games to test a change, put him on top for 5-6 years. Vas also said, re-check your old failed ideas, they might be improvements after all.
Of course testing is the factor that made the top engines strong but for our knowledge about chess we need an alternative explanation because I know nothing more about chess from the explanation of testing when you know more about chess for the explanation about a better way to evaluate chess position. Humans who play chess who want to know why stockfish played a move in many cases so they can also make better moves certainaly need better explanation.
God does not explain itself and, if it did, you wouldn't understand.
