Sven wrote:Problem of not running under cutechess-cli was solved easily (cygwin DLLs had to be copied to the right folder ...). Now I run some tests. As expected, the "onecolor" version makes illegal moves so I took it out.
Jumbo seems too strong for Embla, I even tried an old 0.4.14 version which is roughly 200 Elo points weaker than my current best version. So I'll match Embla versions against each other only.
I usually let it battle against dorpsgek which is around 220 elo stronger.
What do you do for tournament? Gauntlet?
Round robin between four Embla versions: master, master with an attempt to disable LMR (but maybe I did not do it completely right), PVS and PVS with the "bSearchPV=false" statement moved directly behind undoMove.
But I stopped it already after 720 games. The PVS branch, as on Github, appears to be broken, both PVS versions played about 440-460 Elo points (+/- 30) weaker than both master versions. So this is significant enough to stop and say it is broken somehow. Maybe something went wrong when merging? Based on the Github history it seems as if the PVS branch started from an earlier commit of the "lazy-smp" branch (c88edbc) and therefore includes many changes from there (but not the latest ones) while it does not include a considerable number of recent changes from master. I can't imagine that the concepts behind these changes make up for 450 Elo points so I would expect some weird merge problem or similar.
So either GitHub is still not in sync with your local state, or your PVS branch is somehow broken, or my test run is broken (I would be surprised about the latter).
Sven wrote:Problem of not running under cutechess-cli was solved easily (cygwin DLLs had to be copied to the right folder ...). Now I run some tests. As expected, the "onecolor" version makes illegal moves so I took it out.
Jumbo seems too strong for Embla, I even tried an old 0.4.14 version which is roughly 200 Elo points weaker than my current best version. So I'll match Embla versions against each other only.
I usually let it battle against dorpsgek which is around 220 elo stronger.
What do you do for tournament? Gauntlet?
Round robin between four Embla versions: master, master with an attempt to disable LMR (but maybe I did not do it completely right), PVS and PVS with the "bSearchPV=false" statement moved directly behind undoMove.
But I stopped it already after 720 games. The PVS branch, as on Github, appears to be broken, both PVS versions played about 440-460 Elo points (+/- 30) weaker than both master versions. So this is significant enough to stop and say it is broken somehow. Maybe something went wrong when merging? Based on the Github history it seems as if the PVS branch started from an earlier commit of the "lazy-smp" branch (c88edbc) and therefore includes many changes from there (but not the latest ones) while it does not include a considerable number of recent changes from master. I can't imagine that the concepts behind these changes make up for 450 Elo points so I would expect some weird merge problem or similar.
So either GitHub is still not in sync with your local state, or your PVS branch is somehow broken, or my test run is broken (I would be surprised about the latter).
Please ignore any history orders: I develop in svn and occasionally push a new version of a branch to github. So order can be different.
Ok what I did: deleted the branch, recreated it and pvs into master (with lmr included).
I have to finish my tests for today. I think your testing resources are much better than mine. I can only test on two cores at once (with three or more it always becomes unstable) so I can't play many games per hour. Results are inconclusive so far. What I have looks like this:
master-noLMR-withPVS-manually 24 21 21 424
PVS-newBranch-noLMR -1 21 21 400
master-noLMR -3 15 15 822 (has also other games)
That means (but note the high error bars) that all three versions (master without LMR, your current PVS without LMR, and master without LMR and with manually added PVS in the way you did the change) are about equal.
It requires a high number of games to measure the improvement by PVS, I remember that the expectation was about 10-20 Elo. So my test is simply not sufficient for it. But at least I can't confirm your "-50 Elo" result so far. I still believe, after having seen all those changes you did recently, that your measurement might have been influenced by some version control/merge/etc. problem.
flok wrote:Ah that I didn't know. Can you advise one by any chance? I'm currently use the dc-3200.bin polyglot book (from http://rebel13.nl/download/books.html).
I don't know that book, but you'll find dozens of books in polyglot bin format or pgn start files alone with a simple search here in the forum.
It also depends on, which GUI or tool you are using for your test games, if it supports limiting depth you could just stay with what you have and limit it to
something between 6 and 10 moves.
(Fishtest even uses a 2 moves book, but the positions are sometimes a bit weird and probably only make sense for an already very strong program)
A bit of a moot point, as that code sucks anyway: It calls search() twice in a row for the same position. So that a lot of stuff has to be done in duplicate (check testing, move generation...), and almost nothing is learned from the previous call (move ordering).
A bit of a moot point, as that code sucks anyway: It calls search() twice in a row for the same position. So that a lot of stuff has to be done in duplicate (check testing, move generation...), and almost nothing is learned from the previous call (move ordering).
Inherent for any kind of re-search. The two trees being searched for a zero vs. a full window are also very different. The stuff that is repeated (at the current node) is really negligible, main effort is for searching the subtree. "Learning" happens through TT.
