Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

[ThatsIt]

Hi Larry,

i've also a question.
Is MCTS-Komodo different enough in comparision to Komodo without MCTS
to speak of an "independent" engine ?
If so, i will include the MCTS-Version into our 3'+1" ponder=on list and of
course then both versions have to play each other.

Best wishes,
G.S.
(CEGT team)

[lkaufman]

Thanks. Do you set the number of games to run at once (assuming single thread test) to equal the number of cores, or to one less than number of cores? For example, if you have a quad, do you run four games at once or three games at once? Is there a uniform policy on this question?

Best,
Larry

p.s. So far it looks like our best result in your testing is vs. SF, so perhaps I have overestimated a bit due to relying heavily on testing vs. SF. Anyway, if we have to overperform against one engine, we picked the right one!

[ThatsIt]

Update, 900 games are played.

http://cegt.forumieren.com/t1043-testin ... o-12-2-x64

+ 229 to version 12.1.1 now !

Best wishes,
G.S.
(CEGT team)

[ThatsIt]

Hi Larry !

Thanks. Do you set the number of games to run at once (assuming single thread test) to equal the number of cores, or to one less than number of cores? For example, if you have a quad, do you run four games at once or three games at once? Is there a uniform policy on this question?

On both i5s (max. 4 cores) i use max. 3 cores and on the FX8350 (max. 8 cores)
max. 6 cores. But thats only my kind of testing.

btw.:
have you seen my question ?

Best wishes,
G.S.
(CEGT team)

[lkaufman]

Hi Larry,

i've also a question.
Is MCTS-Komodo different enough in comparision to Komodo without MCTS
to speak of an "independent" engine ?
If so, i will include the MCTS-Version into our 3'+1" ponder=on list and of
course then both versions have to play each other.

Best wishes,
G.S.
(CEGT team)

The evaluation function is similar to normal Komodo, with one major difference regarding King Safety. The search is somewhat similar to Lc0. Since Lc0 and normal Komodo are about as different as any two engines can be, I would say that the difference between normal Komodo and Komodo MCTS is quite large, maybe comparable to the differences among Komodo, Stockfish, and Houdini. We will probably split them into separate engines at some point; ChessBase already did that for Komodo 12 and 12.1.1. So yes, I think that is a good idea. In my own testing Komodo MCTS gets around the same performance rating vs. older versions of Stockfish and Komodo; there is no apparent problem with running a normal Komodo vs MCTS Komodo.

[lkaufman]

Hi Larry !

Thanks. Do you set the number of games to run at once (assuming single thread test) to equal the number of cores, or to one less than number of cores? For example, if you have a quad, do you run four games at once or three games at once? Is there a uniform policy on this question?

On both i5s (max. 4 cores) i use max. 3 cores and on the FX8350 (max. 8 cores)
max. 6 cores. But thats only my kind of testing.

btw.:
have you seen my question ?

I missed it before, but I answered it now. We recently concluded that leaving at least one core free for GUI and Windows is important, so we agree on this point.

Best wishes,
G.S.
(CEGT team)

[lkaufman]

Hi Larry,

How much memory is the minimum allowed or used for a test machine?

All my machines have 4 or more GB RAM.

Best wishes,
G.S.
(CEGT team)

I wanted to mention one possible problem, since you imply that at least one of your machines has only 4 GB RAM. If you are testing 3 games at once involving Komodo MCTS, it would be using (256 + 128 + 320) Mb of RAM (adding hash, table memory, and mcts hash) per game, which is 704. The opposing machine would be using at least 256 + ??, let's say at least 300 total. So at least 1 Gb per game. Three games would leave just 1 Gb for operating system, GUI, etc., which I think is not enough. Maybe you only run one or two games on a machine with just 4 Gb RAM, which should be OK, but if you run 3 I think the test might not be valid. Our bugfix version will reduce the default MCTS hash to 128, and you can safely reduce table memory to 64, which together might be enough to run three games on a 4 Gb RAM machine.

[ThatsIt]

Hi Larry,

I wanted to mention one possible problem, since you imply that at least one of your machines has only 4 GB RAM. If you are testing 3 games at once involving Komodo MCTS, it would be using (256 + 128 + 320) Mb of RAM (adding hash, table memory, and mcts hash) per game, which is 704. The opposing machine would be using at least 256 + ??, let's say at least 300 total. So at least 1 Gb per game. Three games would leave just 1 Gb for operating system, GUI, etc., which I think is not enough. Maybe you only run one or two games on a machine with just 4 Gb RAM, which should be OK, but if you run 3 I think the test might not be valid. Our bugfix version will reduce the default MCTS hash to 128, and you can safely reduce table memory to 64, which together might be enough to run three games on a 4 Gb RAM machine.

The i5-2400 with 4GB RAM wasn't involved in MCTS-Komodo-Testing.
This machine is busy currently with:
Monik 2.2.6 x64:
http://cegt.forumieren.com/t1036-testin ... -2-2-6-x64
and
HoiChess 0.22.0 x64:
http://cegt.forumieren.com/t1039-testin ... 0-22-0-x64

Best wishes,
G.S.
(CEGT team)

[ThatsIt]

Update, 1500 games are played.

+ 225 to version 12.1.1 now !

Code: Select all

CEGT 40/4 + 40/4 + ...
...
Komodo 12.1.1 x64 1CPU         3337  +12 -12
Fire 7.1 x64 1CPU              3244  +12 -12
Ethereal 11.00 x64 1CPU        3183  +14 -14
Komodo 12.2 x64 1CPU (MCTS)    3174  +15 -15
Deep Shredder 13 x64 1CPU      3152  + 6 - 6
Booot 6.3 x64 1CPU             3131  +11 -11
...
Komodo 12.1.1 x64 1CPU (MCTS)  2949  +10 -10
...

Best wishes,
G.S.

[Chessqueen]

Hi Larry,

I better make my prediction before any results are posted. I think it will be a bit over 3200 on your scale, I'll say 3210. This would be more than a 250 elo gain from the last release if I am correct. We shall see.

Sure?
You wrote:
"The strength of the MCTS version is vastly improved, up around 200 elo points at blitz levels ..."

But, wait and see ...

Best wishes and many THX for the new version,
G.S.
(CEGT team)

I was looking at CCRL ratings when I said 200, I think your ratings are further spread because CCRL uses bayeselo which contracts the rating scale somewhat. Komodo 12.2 MCTS on one thread has come out slightly ahead of Stockfish 6 and Komodo 9.1 in our tests, that's the basis of my forecast.

I believe that the CCRL rating should be lower by 250 points overall, since there is no based to compare it with human FIDE rating, and it would make it more realistic.