Eman Chimera - hints and impressions

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
pohl4711
Posts: 1241
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Eman Chimera - hints and impressions

Post by pohl4711 » Mon May 18, 2020 10:48 am

https://eman.zohosites.com/files/Eman%2 ... 0v1.10.zip

The tool is awesome, but has some unused potential, to make it even better...A switch from Engine A to B by movenumber is not as good as by number of pieces on the board (in pawns, added to one number). But movenumber is not so bad as it seems, because of Lc0's bad time-management: When playing with increment, lc0 uses the whole basetime until move 60 most of the games. So, a switch to a AB-engine some moves before 60 makes always sense.

As I reported here before, Chimera works fine in FritzGUI 17 (pondering works, too (!)). Now, I can report, it works fine in cutechess-cli, too. I started a testrun for my NBSC-Advanced Armageddon testing vs. Stockfish:
https://www.sp-cc.de/nn-vs-sf-testing.htm

Chimera (first 52 moves Lc0 0.24.1 LS 14.3, then Komodo 14 for the rest of the games (Mention, that 4 moves of each NBSC-opening lines are just for making the short castle of black illegal, so 48 moves are played, only, when Chimera switches to Komodo 14). And Chimera switches 1 or 2 moves (depends on played color) too late to the Engine B, so Lc0 plays the first 50 moves (plus/minus one move)).


My config-file of Chimera (chimera.xml) looks like this:
(Important: You have to change SwitchingOverheadSeconds="1" manually with an editor (default is 30, which is way too much for short timecontrols!!)).
With that configuration, the first 48 of 500 games in my testrun were played without any timelosses, stalls or other problems with cutechess-cli.

Code: Select all

<ChimeraUciConfig SwitchingOverheadSeconds="1" ShowThreads="false" ShowHash="false">
	<Name>chimera</Name>
	<Engine FromMove="0">
		<Exe>C:\Cutechess\Engines\Chimera\lc0.exe</Exe>
		<Arguments />
		<Username />
		<Password />
		<Param Name="Backend">cudnn-fp16</Param>
		<Param Name="NNCacheSize">5000000</Param>
		<Param Name="OutOfOrderEval">False</Param>
		<Param Name="MoveOverheadMs">1000</Param>
	</Engine>
	<Engine FromMove="52">
		<Exe>C:\Cutechess\Engines\Chimera\komodo-14-64bit-bmi2.exe</Exe>
		<Arguments />
		<Username />
		<Password />
		<Param Name="Threads">11</Param>
		<Param Name="Ponder">False</Param>
		<Param Name="Hash">2048</Param>
		<Param Name="Overhead ms">1000</Param>
		<Param Name="Contempt">0</Param>
	</Engine>
</ChimeraUciConfig>
After the first 48 games (very, very early intermediate result!!!!) the score of Chimera is 67.3% vs Stockfish 200418 (a new dev-version from middle of april, around +10 Elo to SF 11)...That is very impressive, because the final result (500 games) of Lc0 0.24.1 LS 14.3 alone in my testings was 60.6%.
And mention, that Komodo 14, which plays from move 50 vs Stockfish is
a) around 100 Elo weaker than Stockfish in ratinglists
b) Stockfish has +50% more thinkingtime in my testruns, in order to get a Leela-Ratio of 1.0: Lc0/Chimera: 2'+1'', SF has 3'+1.5'' thinking-time. That is a clear disadvantage for Chimera, because Komodo uses the CPU (of course), like SF and because of this, SF has +50% more time than Komodo 14, when Komodo 14 takes over at move 50...

Final result in 2-3 days. Stay tuned!

(When the testrun is finished, I will try the same run again, with Chimera switching to Komodo 14 10 moves earlier...Closer to the movenumber, K.Laskos suggests (35). Lets see, what is better)

With Chimera, other engine-setups are possible (and interesting), of course:
What about switching from one lc0 to another, when the second lc0 uses an endgame NN ?
https://github.com/dkappe/leela-chess-w ... ndgame-Net
Could be interesting, too....

dkappe
Posts: 371
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: Eman Chimera - hints and impressions

Post by dkappe » Mon May 18, 2020 8:06 pm

I’ve been waiting for this net switching capability to release Little Ender and Medium Ender in lc0 format. Will do so shortly.

Although they were trained with 18 man positions or less, they seem to have a little more tactical blindness than full nets. I’d recommend 12 man as the switching point.

dkappe
Posts: 371
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: Eman Chimera - hints and impressions

Post by dkappe » Mon May 18, 2020 8:23 pm


jjoshua2
Posts: 67
Joined: Sat Mar 10, 2018 5:16 am

Re: Eman Chimera - hints and impressions

Post by jjoshua2 » Mon May 18, 2020 8:28 pm

Awesome! Also see:
Version 2.00 - April 17, 2020
Added ability to switch engines based on remaining piece count (including pawns)

pohl4711
Posts: 1241
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Eman Chimera - hints and impressions

Post by pohl4711 » Tue May 19, 2020 9:43 am

I aborted my testrun. After 200 games, Chimera (first 52 moves Lc0 0.24.1 LS 14.3, then Komodo 14 for the rest of the games) had exactly the same score than Lc0 LS 14.3 had as final result. Perhaps Komodo is too weak?
Because of the release of Chimera 2.00, I decided to restart the testrun with Chimera (Lc0 0.24.1 LS 14.3 and switching to Stockfish 200418, when 16 or less pieces are on the board)(Stockfish 200418 is the same SF, which is the opponent in my testrun).
Hopefully, that will give a better result.

User avatar
Laskos
Posts: 10197
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Eman Chimera - hints and impressions

Post by Laskos » Tue May 19, 2020 11:02 am

pohl4711 wrote:
Tue May 19, 2020 9:43 am
I aborted my testrun. After 200 games, Chimera (first 52 moves Lc0 0.24.1 LS 14.3, then Komodo 14 for the rest of the games) had exactly the same score than Lc0 LS 14.3 had as final result. Perhaps Komodo is too weak?
Because of the release of Chimera 2.00, I decided to restart the testrun with Chimera (Lc0 0.24.1 LS 14.3 and switching to Stockfish 200418, when 16 or less pieces are on the board)(Stockfish 200418 is the same SF, which is the opponent in my testrun).
Hopefully, that will give a better result.
I have plenty of good results at short TC in Cutechess-CLI, the best seems to switch at move 36 to SF11 (4 threads) ||| + 60 Elo points.
Lc0 used is v25.1.
Strangely enough, material condition doesn't fare well, but needs to be investigated further. Both move and material conditions combined might work well.

Code: Select all


=======================================
15s+0.25s

Basis:

GM_4moves openings:

Score of SV_384x30_3200 vs SF_11: 41 - 13 - 146  [0.570] 200
Elo difference: 48.96 +/- 24.58
Finished match
============================



Switch from SV_384x30_3200 to SF11

=============================
Move number
GM_4moves openings:

Score of Chimera_move25 vs SF_11: 27 - 4 - 69  [0.615] 100
Elo difference: 81.37 +/- 37.20
Finished match

Score of Chimera_move35 vs SF_11: 34 - 3 - 63  [0.655] 100
Elo difference: 111.37 +/- 40.67
Finished match

Score of Chimera_move45 vs SF_11: 33 - 8 - 59  [0.625] 100
Elo difference: 88.74 +/- 43.26
Finished match


TCEC9 openings

Score of Chimera_move35 vs SF_11: 36 - 7 - 57  [0.645] 100
Elo difference: 103.73 +/- 43.96
Finished match
===========================
Material
GM_4moves openings:

Score of Chimera_pieces16 vs SF_11: 39 - 21 - 140  [0.545] 200
Elo difference: 31.35 +/- 26.24
Finished match

Score of Chimera_pieces20 vs SF_11: 56 - 23 - 121  [0.583] 200
Elo difference: 57.86 +/- 30.06
Finished match

pohl4711
Posts: 1241
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Eman Chimera - hints and impressions

Post by pohl4711 » Tue May 19, 2020 12:08 pm

Laskos wrote:
Tue May 19, 2020 11:02 am
pohl4711 wrote:
Tue May 19, 2020 9:43 am
I aborted my testrun. After 200 games, Chimera (first 52 moves Lc0 0.24.1 LS 14.3, then Komodo 14 for the rest of the games) had exactly the same score than Lc0 LS 14.3 had as final result. Perhaps Komodo is too weak?
Because of the release of Chimera 2.00, I decided to restart the testrun with Chimera (Lc0 0.24.1 LS 14.3 and switching to Stockfish 200418, when 16 or less pieces are on the board)(Stockfish 200418 is the same SF, which is the opponent in my testrun).
Hopefully, that will give a better result.
I have plenty of good results at short TC in Cutechess-CLI, the best seems to switch at move 36 to SF11 (4 threads) ||| + 60 Elo points.
Lc0 used is v25.1.
Strangely enough, material condition doesn't fare well, but needs to be investigated further. Both move and material conditions combined might work well.
Both move and material conditions are impossible in Chimera. You have to choose one option.

In my tests, cutechess-cli ends all games, when a 5man Syzygy is on the board. Perhaps, thats a reason for lc0 playing with more success (without Chimera)?!?

User avatar
Laskos
Posts: 10197
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Eman Chimera - hints and impressions

Post by Laskos » Tue May 19, 2020 12:25 pm

pohl4711 wrote:
Tue May 19, 2020 12:08 pm
Laskos wrote:
Tue May 19, 2020 11:02 am
pohl4711 wrote:
Tue May 19, 2020 9:43 am
I aborted my testrun. After 200 games, Chimera (first 52 moves Lc0 0.24.1 LS 14.3, then Komodo 14 for the rest of the games) had exactly the same score than Lc0 LS 14.3 had as final result. Perhaps Komodo is too weak?
Because of the release of Chimera 2.00, I decided to restart the testrun with Chimera (Lc0 0.24.1 LS 14.3 and switching to Stockfish 200418, when 16 or less pieces are on the board)(Stockfish 200418 is the same SF, which is the opponent in my testrun).
Hopefully, that will give a better result.
I have plenty of good results at short TC in Cutechess-CLI, the best seems to switch at move 36 to SF11 (4 threads) ||| + 60 Elo points.
Lc0 used is v25.1.
Strangely enough, material condition doesn't fare well, but needs to be investigated further. Both move and material conditions combined might work well.
Both move and material conditions are impossible in Chimera. You have to choose one option.

In my tests, cutechess-cli ends all games, when a 5man Syzygy is on the board. Perhaps, thats a reason for lc0 playing with more success (without Chimera)?!?
Not the main reason, but a reason. I don't adjudicate at all.

User avatar
Laskos
Posts: 10197
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Eman Chimera - hints and impressions

Post by Laskos » Tue May 19, 2020 12:27 pm

pohl4711 wrote:
Tue May 19, 2020 12:08 pm
Laskos wrote:
Tue May 19, 2020 11:02 am
pohl4711 wrote:
Tue May 19, 2020 9:43 am
I aborted my testrun. After 200 games, Chimera (first 52 moves Lc0 0.24.1 LS 14.3, then Komodo 14 for the rest of the games) had exactly the same score than Lc0 LS 14.3 had as final result. Perhaps Komodo is too weak?
Because of the release of Chimera 2.00, I decided to restart the testrun with Chimera (Lc0 0.24.1 LS 14.3 and switching to Stockfish 200418, when 16 or less pieces are on the board)(Stockfish 200418 is the same SF, which is the opponent in my testrun).
Hopefully, that will give a better result.
I have plenty of good results at short TC in Cutechess-CLI, the best seems to switch at move 36 to SF11 (4 threads) ||| + 60 Elo points.
Lc0 used is v25.1.
Strangely enough, material condition doesn't fare well, but needs to be investigated further. Both move and material conditions combined might work well.
Both move and material conditions are impossible in Chimera. You have to choose one option.
Ferdy is making a tool which can combine them. I will now look into it, at first try it didn't work, Chimera always exited the games.

Joerg Oster
Posts: 727
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: Eman Chimera - hints and impressions

Post by Joerg Oster » Tue May 19, 2020 1:08 pm

Laskos wrote:
Tue May 19, 2020 12:27 pm
pohl4711 wrote:
Tue May 19, 2020 12:08 pm
Laskos wrote:
Tue May 19, 2020 11:02 am
pohl4711 wrote:
Tue May 19, 2020 9:43 am
I aborted my testrun. After 200 games, Chimera (first 52 moves Lc0 0.24.1 LS 14.3, then Komodo 14 for the rest of the games) had exactly the same score than Lc0 LS 14.3 had as final result. Perhaps Komodo is too weak?
Because of the release of Chimera 2.00, I decided to restart the testrun with Chimera (Lc0 0.24.1 LS 14.3 and switching to Stockfish 200418, when 16 or less pieces are on the board)(Stockfish 200418 is the same SF, which is the opponent in my testrun).
Hopefully, that will give a better result.
I have plenty of good results at short TC in Cutechess-CLI, the best seems to switch at move 36 to SF11 (4 threads) ||| + 60 Elo points.
Lc0 used is v25.1.
Strangely enough, material condition doesn't fare well, but needs to be investigated further. Both move and material conditions combined might work well.
Both move and material conditions are impossible in Chimera. You have to choose one option.
Ferdy is making a tool which can combine them. I will now look into it, at first try it didn't work, Chimera always exited the games.
In general, this looks like a feature to be implemented/provided by the GUI/cli-interface.
Jörg Oster

Post Reply