Adam Hair wrote:I used these time controls because I needed to know the change in Elo with thinking times in this range. I do not expect that the Elo increase holds for longer time controls. But this is what I had in mind when I said "shorter thinking times". However, due to search improvements and hardware improvements, I would assume that the depths reached at these time controls may not be inferior to those reached in the tests that determined the 50 to 70 Elo increase per doubling. Which I assumed occurred before How Computers Play Chess was published, where this "fact" was stated.
When the article was published EBFs were much higher at those depths (which you effectively test with your ultra-short TCs) than today. That's one reason you should not test at those TC's (at longer TC's average EBF is a much better representation of the strength of the program).
Second reason is that at ultra-short TCs (where time per move is much lower than 1sec) a time you dedicate to a move is much smaller than the real time engine spends searching. That's because the overhead of interface, time management, uncertainty of time measurement, SMP implementation, etc. plays much higher % in total time per move engine has. This reduces when time per move is increased and when time per move goes over 1 sec it becomes practically negligible.
By the way, I think that extremely fast time controls are used in engine tuning in order to accumulate enough games in a reasonably amount of time in order to achieve some statistical significance. At least that is the rationale most authors use.
It's both. In order to see if the change is positive or not (to be included in the release) you need that absolute value of a change is higher than the variance of the measurement of that value. Variance you reduce by higher number of games (which is faster with ultra-short TCs) and absolute value you increase (exaggerate) with ultra-short TCs.
In my tests, each engine played every other engine at the same time control. Based on those results, the opponents for a certain engine at a certain time control were determined. Those opponents were in a +/- 100 Elo interval from the intermediate rating for that engine/TC.
That is wrong setup.
An engine that you test at different TCs has to have same opponents all the time and its opponents need to have the same thinking time in all the matches, i.e. thinking time of the opponent has to be independent of the thinking time of the tested engine.
Therefore I suggest you select the pool of 5 test engines (with the strength range from the fastest TC of tested engine to slowest TC of tested engine) and test then against a single engine that plays at 3 TCs, for example at 1'+1'', 2'+2'' and 4'+4''.
P.S. All the tests should be done on a single core per engine (no SMP involved).
Very well. Instead of fruitless debate, I will perform this test to your specifications. I will determine the engines to test and start a thread to keep interested people informed.
I just want to announce ahead of time that the most likely results will be intermediate to my previous results and your prediction.
Especially when it's about something many programmers already know and you apparently don't
In the 80's the branch factor was about 4.0 and the estimated elo gain when doubling speed already was roughly 50-70 elo because an engine became ½ ply deeper. Nowadays with a branch factor of >= 1.5 <= 2.0 a doubling of speed gives a full ply to 1½ ply deeper search, hence the elo gain goes up as well.
Can you suggest somewhere where I can read about this? I tried wikipedia, but no joy.
Branching Factors have always confused me because they seem so incredibly low (I would have imagined a factor of 10, a factor of 4 seems barely plausible but 1.5 seems ridiculous or witchcraft)
I don't know much about this - what articles are good, please?
1) There is alpha-beta, which not chess-specific. It's a lossless method to cut trees. The effectivity depends on the ordering of the branches. That's why move-ordering in chess is so important.
I have never heard about a chess program NOT using alpha-beta. Anyone?
2) Then there is a lof of pruning (e.g. Null-Move). This is always lossy. So if you prune a lot, you got very low branch-factors, but you oversee a lot of nice moves, too. That's the price.
Especially when it's about something many programmers already know and you apparently don't
In the 80's the branch factor was about 4.0 and the estimated elo gain when doubling speed already was roughly 50-70 elo because an engine became ½ ply deeper. Nowadays with a branch factor of >= 1.5 <= 2.0 a doubling of speed gives a full ply to 1½ ply deeper search, hence the elo gain goes up as well.
Can you suggest somewhere where I can read about this? I tried wikipedia, but no joy.
Branching Factors have always confused me because they seem so incredibly low (I would have imagined a factor of 10, a factor of 4 seems barely plausible but 1.5 seems ridiculous or witchcraft)
I don't know much about this - what articles are good, please?
1) There is alpha-beta, which not chess-specific. It's a lossless method to cut trees. The effectivity depends on the ordering of the branches. That's why move-ordering in chess is so important.
I've done some reading on Alpha-Beta pruning. There are two things I don't get:
i) How is Alpha-Beta pruning 'lossless'? Anything which prunes must take a risk. Just because you decide that a move is terrible at ply 7 doesn't mean that at ply 23 it can't be shown to be good again.
ii) How do chess engines decide to terminate their search at ply x? If they run their search and conclude the move 10 Nf5 is bad (presumably by running the evaluation code on Nf5) and stop there it could be that searching just one more ply would show 10 Nf5 was actually good.
Especially when it's about something many programmers already know and you apparently don't
In the 80's the branch factor was about 4.0 and the estimated elo gain when doubling speed already was roughly 50-70 elo because an engine became ½ ply deeper. Nowadays with a branch factor of >= 1.5 <= 2.0 a doubling of speed gives a full ply to 1½ ply deeper search, hence the elo gain goes up as well.
Can you suggest somewhere where I can read about this? I tried wikipedia, but no joy.
Branching Factors have always confused me because they seem so incredibly low (I would have imagined a factor of 10, a factor of 4 seems barely plausible but 1.5 seems ridiculous or witchcraft)
I don't know much about this - what articles are good, please?
1) There is alpha-beta, which not chess-specific. It's a lossless method to cut trees. The effectivity depends on the ordering of the branches. That's why move-ordering in chess is so important.
I've done some reading on Alpha-Beta pruning. There are two things I don't get:
i) How is Alpha-Beta pruning 'lossless'? Anything which prunes must take a risk. Just because you decide that a move is terrible at ply 7 doesn't mean that at ply 23 it can't be shown to be good again.
ii) How do chess engines decide to terminate their search at ply x? If they run their search and conclude the move 10 Nf5 is bad (presumably by running the evaluation code on Nf5) and stop there it could be that searching just one more ply would show 10 Nf5 was actually good.
i) Alpha-beta pruning does not permanently prune a move, it prunes it for a given depth. And yes, it's absolutely lossless, as proven by any decent AI book and probably wikipedia too.
ii) Read up on "iterative deepening". Basically the programs keep increasing the depth of their search until they run out of time (and they can vary the time per move a bit depending on whether their score is improving and related factors).
I think that human masters (and by master I mean 2400 FIDE) are far better than chess engines by at least a 1000 Elo points. And I'm not even about Advanced Chess or Correspondence GMs.
Chess engines clearly don't know to play chess very well and not even the best ones.
Houdini 3 Pro was beaten 10-0 by Ivanov Borislav and Ivanov is not even rated 2700 at FIDE. This means that even the best commercial programs like Houdini MP can be beaten relatively easily by someone who is not even rated 2700 in FIDE.
This is why I think that this forum is not such a good place for a weak player to be able to improve at chess because chess engines clearly don't stand a chance against human Grandmasters and especially human Correspondence Grandmasters.
Uri wrote:I think that human masters (and by master I mean 2400 FIDE) are far better than chess engines by at least a 1000 Elo points. And I'm not even about Advanced Chess or Correspondence GMs.
Chess engines clearly don't know to play chess very well and not even the best ones.
Houdini 3 Pro was beaten 10-0 by Ivanov Borislav and Ivanov is not even rated 2700 at FIDE. This means that even the best commercial programs like Houdini MP can be beaten relatively easily by someone who is not even rated 2700 in FIDE.
This is why I think that this forum is not such a good place for a weak player to be able to improve at chess because chess engines clearly don't stand a chance against human Grandmasters and especially human Correspondence Grandmasters.
