Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
ThatsIt
Posts: 837
Joined: Thu Mar 09, 2006 1:11 pm
Contact:

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by ThatsIt » Thu Nov 08, 2018 8:09 am

Hi Larry,

i've also a question.
Is MCTS-Komodo different enough in comparision to Komodo without MCTS
to speak of an "independent" engine ?
If so, i will include the MCTS-Version into our 3'+1" ponder=on list and of
course then both versions have to play each other.

Best wishes,
G.S.
(CEGT team)

lkaufman
Posts: 3190
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by lkaufman » Thu Nov 08, 2018 5:03 pm

Thanks. Do you set the number of games to run at once (assuming single thread test) to equal the number of cores, or to one less than number of cores? For example, if you have a quad, do you run four games at once or three games at once? Is there a uniform policy on this question?

Best,
Larry

p.s. So far it looks like our best result in your testing is vs. SF, so perhaps I have overestimated a bit due to relying heavily on testing vs. SF. Anyway, if we have to overperform against one engine, we picked the right one!
Komodo rules!

ThatsIt
Posts: 837
Joined: Thu Mar 09, 2006 1:11 pm
Contact:

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by ThatsIt » Thu Nov 08, 2018 6:20 pm

Update, 900 games are played.

http://cegt.forumieren.com/t1043-testin ... o-12-2-x64

+ 229 to version 12.1.1 now !

Best wishes,
G.S.
(CEGT team)

ThatsIt
Posts: 837
Joined: Thu Mar 09, 2006 1:11 pm
Contact:

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by ThatsIt » Thu Nov 08, 2018 6:27 pm

Hi Larry !
lkaufman wrote:
Thu Nov 08, 2018 5:03 pm
Thanks. Do you set the number of games to run at once (assuming single thread test) to equal the number of cores, or to one less than number of cores? For example, if you have a quad, do you run four games at once or three games at once? Is there a uniform policy on this question?
On both i5s (max. 4 cores) i use max. 3 cores and on the FX8350 (max. 8 cores)
max. 6 cores. But thats only my kind of testing.

btw.:
have you seen my question ?

Best wishes,
G.S.
(CEGT team)

lkaufman
Posts: 3190
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by lkaufman » Thu Nov 08, 2018 7:38 pm

ThatsIt wrote:
Thu Nov 08, 2018 8:09 am
Hi Larry,

i've also a question.
Is MCTS-Komodo different enough in comparision to Komodo without MCTS
to speak of an "independent" engine ?
If so, i will include the MCTS-Version into our 3'+1" ponder=on list and of
course then both versions have to play each other.

Best wishes,
G.S.
(CEGT team)
The evaluation function is similar to normal Komodo, with one major difference regarding King Safety. The search is somewhat similar to Lc0. Since Lc0 and normal Komodo are about as different as any two engines can be, I would say that the difference between normal Komodo and Komodo MCTS is quite large, maybe comparable to the differences among Komodo, Stockfish, and Houdini. We will probably split them into separate engines at some point; ChessBase already did that for Komodo 12 and 12.1.1. So yes, I think that is a good idea. In my own testing Komodo MCTS gets around the same performance rating vs. older versions of Stockfish and Komodo; there is no apparent problem with running a normal Komodo vs MCTS Komodo.
Komodo rules!

lkaufman
Posts: 3190
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by lkaufman » Thu Nov 08, 2018 7:41 pm

ThatsIt wrote:
Thu Nov 08, 2018 6:27 pm
Hi Larry !
lkaufman wrote:
Thu Nov 08, 2018 5:03 pm
Thanks. Do you set the number of games to run at once (assuming single thread test) to equal the number of cores, or to one less than number of cores? For example, if you have a quad, do you run four games at once or three games at once? Is there a uniform policy on this question?
On both i5s (max. 4 cores) i use max. 3 cores and on the FX8350 (max. 8 cores)
max. 6 cores. But thats only my kind of testing.

btw.:
have you seen my question ?

I missed it before, but I answered it now. We recently concluded that leaving at least one core free for GUI and Windows is important, so we agree on this point.

Best wishes,
G.S.
(CEGT team)
Komodo rules!

lkaufman
Posts: 3190
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by lkaufman » Thu Nov 08, 2018 8:10 pm

ThatsIt wrote:
Thu Nov 08, 2018 7:56 am
Hi Larry,
lkaufman wrote:
Wed Nov 07, 2018 7:43 pm
How much memory is the minimum allowed or used for a test machine?
All my machines have 4 or more GB RAM.

Best wishes,
G.S.
(CEGT team)
I wanted to mention one possible problem, since you imply that at least one of your machines has only 4 GB RAM. If you are testing 3 games at once involving Komodo MCTS, it would be using (256 + 128 + 320) Mb of RAM (adding hash, table memory, and mcts hash) per game, which is 704. The opposing machine would be using at least 256 + ??, let's say at least 300 total. So at least 1 Gb per game. Three games would leave just 1 Gb for operating system, GUI, etc., which I think is not enough. Maybe you only run one or two games on a machine with just 4 Gb RAM, which should be OK, but if you run 3 I think the test might not be valid. Our bugfix version will reduce the default MCTS hash to 128, and you can safely reduce table memory to 64, which together might be enough to run three games on a 4 Gb RAM machine.
Komodo rules!

ThatsIt
Posts: 837
Joined: Thu Mar 09, 2006 1:11 pm
Contact:

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by ThatsIt » Fri Nov 09, 2018 6:16 am

Hi Larry,
lkaufman wrote:
Thu Nov 08, 2018 8:10 pm
I wanted to mention one possible problem, since you imply that at least one of your machines has only 4 GB RAM. If you are testing 3 games at once involving Komodo MCTS, it would be using (256 + 128 + 320) Mb of RAM (adding hash, table memory, and mcts hash) per game, which is 704. The opposing machine would be using at least 256 + ??, let's say at least 300 total. So at least 1 Gb per game. Three games would leave just 1 Gb for operating system, GUI, etc., which I think is not enough. Maybe you only run one or two games on a machine with just 4 Gb RAM, which should be OK, but if you run 3 I think the test might not be valid. Our bugfix version will reduce the default MCTS hash to 128, and you can safely reduce table memory to 64, which together might be enough to run three games on a 4 Gb RAM machine.
The i5-2400 with 4GB RAM wasn't involved in MCTS-Komodo-Testing.
This machine is busy currently with:
Monik 2.2.6 x64:
http://cegt.forumieren.com/t1036-testin ... -2-2-6-x64
and
HoiChess 0.22.0 x64:
http://cegt.forumieren.com/t1039-testin ... 0-22-0-x64

Best wishes,
G.S.
(CEGT team)

ThatsIt
Posts: 837
Joined: Thu Mar 09, 2006 1:11 pm
Contact:

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by ThatsIt » Fri Nov 09, 2018 9:09 pm

Update, 1500 games are played.

+ 225 to version 12.1.1 now !

Code: Select all

CEGT 40/4 + 40/4 + ...
...
Komodo 12.1.1 x64 1CPU         3337  +12 -12
Fire 7.1 x64 1CPU              3244  +12 -12
Ethereal 11.00 x64 1CPU        3183  +14 -14
Komodo 12.2 x64 1CPU (MCTS)    3174  +15 -15
Deep Shredder 13 x64 1CPU      3152  + 6 - 6
Booot 6.3 x64 1CPU             3131  +11 -11
...
Komodo 12.1.1 x64 1CPU (MCTS)  2949  +10 -10
...

Best wishes,
G.S.

Chessqueen
Posts: 134
Joined: Wed Sep 05, 2018 12:16 am
Full name: Nancy M Pichardo

Re: Komodo 12.2 x64 (MCTS) @ CEGT 40/4 + ...

Post by Chessqueen » Fri Nov 09, 2018 10:40 pm

lkaufman wrote:
Tue Nov 06, 2018 9:35 pm
ThatsIt wrote:
Tue Nov 06, 2018 8:56 pm
Hi Larry,
lkaufman wrote:
Tue Nov 06, 2018 6:57 pm
I better make my prediction before any results are posted. I think it will be a bit over 3200 on your scale, I'll say 3210. This would be more than a 250 elo gain from the last release if I am correct. We shall see.
Sure?
You wrote:
"The strength of the MCTS version is vastly improved, up around 200 elo points at blitz levels ..."

But, wait and see ...

Best wishes and many THX for the new version,
G.S.
(CEGT team)
I was looking at CCRL ratings when I said 200, I think your ratings are further spread because CCRL uses bayeselo which contracts the rating scale somewhat. Komodo 12.2 MCTS on one thread has come out slightly ahead of Stockfish 6 and Komodo 9.1 in our tests, that's the basis of my forecast.
I believe that the CCRL rating should be lower by 250 points overall, since there is no based to compare it with human FIDE rating, and it would make it more realistic.

Post Reply