hgm wrote:I always assume Elo = 100*ln(speed). That would give you about 70 Elo per doubling. But the exact number is engine-dependent. Engines with a poor move ordering benifit less than those with a good one.
I have a more definitive answer now. If you are running fast games at low depths the constant 100 should be sometime on the order of 150.
the 100 value that hgm gives is probably pretty accurate at normal time controls for strong programs. As the programs play stronger and stronger and search deeper you should expect to lower that value.
This is another way of saying that you should expect diminishing returns with each doubling of effort, but the decline is very gradual.
You can probably also assume that this number drops to zero as you approach perfect play.
Yes, you can assume that it drops to zero. A Calculus way
of stating it would be:
the limit of improvement approaches 0 as the speed of the
program approaches infinity.
Example: you have a program that plays perfectly because
it can see from first move to end of game. If you make the program
10 times faster the program can not play any better because it
played perfectly before the speed up.
Also, it is possible to have a program rated around 1700 (arbitrary number)
that doesn't improve even with a 16x speed up. In such a case,
there exist a major bug holding it back. Fix the bug and get a
major strength increase without speed up. Also, a very poorly
tuned eval will have the same effect as a major bug.
very poorly tuned eval can cause the program to play weaker but
I doubt if it can cause the program to earn nothing from being faster.
Note that in theory even with poor evaluation the program may play perfectly if it searches deep enough.
Uri
In theory, it is possible to have a program that may never play perfectly even searching an infinite time. For instance, a bug that may make you resign one game every other hundred games would be the case. The program will never get more than 99% no matter how much it searches. It may not be a bug, but just an evaluation hole etc.
hgm wrote:I always assume Elo = 100*ln(speed). That would give you about 70 Elo per doubling. But the exact number is engine-dependent. Engines with a poor move ordering benifit less than those with a good one.
Pradu and me got 113 ELO per doubling speed for Dirty . This is a high value, but matches with the general behaviour of the engine, it plays much better at long tc for example. Adjust to the logarithmic curve was pretty accurate . It was self-play though .
hgm wrote:I always assume Elo = 100*ln(speed). That would give you about 70 Elo per doubling. But the exact number is engine-dependent. Engines with a poor move ordering benifit less than those with a good one.
Pradu and me got 113 ELO per doubling speed for Dirty . This is a high value, but matches with the general behaviour of the engine, it plays much better at long tc for example. Adjust to the logarithmic curve was pretty accurate . It was self-play though .
If you are getting 113 in general, you have the strongest program in the world.
But 113 is not a high value at fast time controls. At searches less than 10 ply, it should be much higher than 70 and your value might even be low, but you didn't mention which time control you used.
hgm wrote:I always assume Elo = 100*ln(speed). That would give you about 70 Elo per doubling. But the exact number is engine-dependent. Engines with a poor move ordering benifit less than those with a good one.
Pradu and me got 113 ELO per doubling speed for Dirty . This is a high value, but matches with the general behaviour of the engine, it plays much better at long tc for example. Adjust to the logarithmic curve was pretty accurate . It was self-play though .
If you are getting 113 in general, you have the strongest program in the world.
But 113 is not a high value at fast time controls. At searches less than 10 ply, it should be much higher than 70 and your value might even be low, but you didn't mention which time control you used.
Of course it was blitz, 40/1. A 200 subrounds Round Robin, with 5 instances of the same engine version playing with distinct % of time per move limits. Pradu has the graphics somewhere ...
Note that self-play in general tends to find a larger number, because the trees look so similar, so it is almost impossible for the engine with the smaller tree to surprise the one with the bigger tree. With very differently shaped trees one of them really has to be a lot bigger than the other to totally eclipse it.
hgm wrote:I always assume Elo = 100*ln(speed). That would give you about 70 Elo per doubling. But the exact number is engine-dependent. Engines with a poor move ordering benifit less than those with a good one.
I have a more definitive answer now. If you are running fast games at low depths the constant 100 should be sometime on the order of 150.
the 100 value that hgm gives is probably pretty accurate at normal time controls for strong programs. As the programs play stronger and stronger and search deeper you should expect to lower that value.
This is another way of saying that you should expect diminishing returns with each doubling of effort, but the decline is very gradual.
You can probably also assume that this number drops to zero as you approach perfect play.
Yes, you can assume that it drops to zero. A Calculus way
of stating it would be:
the limit of improvement approaches 0 as the speed of the
program approaches infinity.
Example: you have a program that plays perfectly because
it can see from first move to end of game. If you make the program
10 times faster the program can not play any better because it
played perfectly before the speed up.
Also, it is possible to have a program rated around 1700 (arbitrary number)
that doesn't improve even with a 16x speed up. In such a case,
there exist a major bug holding it back. Fix the bug and get a
major strength increase without speed up. Also, a very poorly
tuned eval will have the same effect as a major bug.
very poorly tuned eval can cause the program to play weaker but
I doubt if it can cause the program to earn nothing from being faster.
Note that in theory even with poor evaluation the program may play perfectly if it searches deep enough.
Uri
In theory, it is possible to have a program that may never play perfectly even searching an infinite time. For instance, a bug that may make you resign one game every other hundred games would be the case. The program will never get more than 99% no matter how much it searches. It may not be a bug, but just an evaluation hole etc.
Miguel
I did not talk about bugs but about very poor evaluation function.
I assume that the evaluation function return mate for mate positions and return draw for draw by repetition or by the fifty move rule or by stalemate.
not doing it is considered by me as a bug.
I did not see a practical example for evaluation function that performs worse at longer time control(note that I assume that the evaluation function is not dependent on previous search because otherwise it may be easy to design evaluation that changes from accurate evaluation in the first second to bad evaluation in later iterations).
I believe that even if you simply only change the sign of your evaluation function at all iterations(except mate positions) you will get slightly better results at longer time control but I did not test it.
hgm wrote:I always assume Elo = 100*ln(speed). That would give you about 70 Elo per doubling. But the exact number is engine-dependent. Engines with a poor move ordering benifit less than those with a good one.
I have a more definitive answer now. If you are running fast games at low depths the constant 100 should be sometime on the order of 150.
the 100 value that hgm gives is probably pretty accurate at normal time controls for strong programs. As the programs play stronger and stronger and search deeper you should expect to lower that value.
This is another way of saying that you should expect diminishing returns with each doubling of effort, but the decline is very gradual.
You can probably also assume that this number drops to zero as you approach perfect play.
Yes, you can assume that it drops to zero. A Calculus way
of stating it would be:
the limit of improvement approaches 0 as the speed of the
program approaches infinity.
Example: you have a program that plays perfectly because
it can see from first move to end of game. If you make the program
10 times faster the program can not play any better because it
played perfectly before the speed up.
Also, it is possible to have a program rated around 1700 (arbitrary number)
that doesn't improve even with a 16x speed up. In such a case,
there exist a major bug holding it back. Fix the bug and get a
major strength increase without speed up. Also, a very poorly
tuned eval will have the same effect as a major bug.
very poorly tuned eval can cause the program to play weaker but
I doubt if it can cause the program to earn nothing from being faster.
Note that in theory even with poor evaluation the program may play perfectly if it searches deep enough.
Uri
In theory, it is possible to have a program that may never play perfectly even searching an infinite time. For instance, a bug that may make you resign one game every other hundred games would be the case. The program will never get more than 99% no matter how much it searches. It may not be a bug, but just an evaluation hole etc.
Miguel
I did not talk about bugs but about very poor evaluation function.
I assume that the evaluation function return mate for mate positions and return draw for draw by repetition or by the fifty move rule or by stalemate.
not doing it is considered by me as a bug.
I did not see a practical example for evaluation function that performs worse at longer time control(note that I assume that the evaluation function is not dependent on previous search because otherwise it may be easy to design evaluation that changes from accurate evaluation in the first second to bad evaluation in later iterations).
I believe that even if you simply only change the sign of your evaluation function at all iterations(except mate positions) you will get slightly better results at longer time control but I did not test it.
Uri
how can that _possibly_ happen? Deeper searches mean you intentionally give up _more_ material or positional advantage if the evaluation sign is backward. No way you can score better, and this is absolutely trivial to test...
I can explain how it can happen and how deeper search can help you to avoid bad moves.
If you search deeper then you see that if you sacrifice your queen then your opponent can force mate against you so you avoid sacrificing the queen.
Another option is that you see that if you sacrifice your queen and force the opponent to capture it then your opponent can use the fact that he has more pieces to sacrifice both the queen and the rook and force you to capture them so you avoid sacrificing the queen.
I did not test it and of course I can be wrong but my guess is that deeper search is going to be slightly better even if you only change the sign of your evaluation.
I can add that I guess that with deep search you may practically not sacrifice your queen because you will prefer to force doing it later when later you are going to avoid it for the same reason so practically you will never sacrifice your queen.
The reason is that sacrificing the queen in the first ply allow your opponent to force something later(even if it is only forcing you to capture the rook) and sacrificing the queen later lead to the same result but the fact that you have to capture the rook after you lose the queen is too deep for the computer to see so practically the computer may think that you lose a full queen by playing a move that does not allow the opponent to capture immediately(that is better than losing queen for a rook).
.
