Trying out the Leela Hybrid engine (NN with AB)

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
KillerDucky
Posts: 12
Joined: Fri Mar 23, 2018 1:31 am

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by KillerDucky » Tue Feb 26, 2019 2:25 am

@pohl4711: Thanks for the logfile, I was able to spot the problem with it! Sorry I didn't test this myself. :oops: It happens with v0.4 if you do not use Syzygy. Fixed in v0.5!

https://github.com/killerducky/lc0/rele ... -fish-v0.5

* Fix crash in v0.4 when Syzygy is not used.
* Stop the AuxEngine when lc0 is done searching.
* This prevents us from cheating by thinking into opponent's time for the last AuxEngine search.
* The cheating was only minor, on my machine ~20 AuxEngine evals are done per move in ~70ms each. So AuxEngine would think an extra 35ms per move on average.
* Fix off by one error in AuxEngineFollowPvDepth.
* Setting this to 1 is supposed to only update the first move, but it would update 2 moves.
* Also update default from 3 to 4 so the behavior is the same as previous releases.
* Add some more logging stats

Code: Select all

auxengine_queue_ size 252 Average duration 72ms Number of evals 23 Number of updates 64

User avatar
sincerelybrent
Posts: 47
Joined: Fri Feb 15, 2019 10:41 pm
Full name: Brent M

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by sincerelybrent » Tue Feb 26, 2019 3:22 am

KillerDucky wrote:
Tue Feb 26, 2019 2:25 am
...It happens with v0.4 if you do not use Syzygy. Fixed in v0.5!
That's why it worked for me. Nice catch, KillerDucky, and thanks for the update.

User avatar
MikeB
Posts: 4053
Joined: Thu Mar 09, 2006 5:34 am
Location: Pen Argyl, Pennsylvania

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by MikeB » Tue Feb 26, 2019 6:24 am

KillerDucky wrote:
Tue Feb 26, 2019 2:25 am
@pohl4711: Thanks for the logfile, I was able to spot the problem with it! Sorry I didn't test this myself. :oops: It happens with v0.4 if you do not use Syzygy. Fixed in v0.5!

https://github.com/killerducky/lc0/rele ... -fish-v0.5

* Fix crash in v0.4 when Syzygy is not used.
* Stop the AuxEngine when lc0 is done searching.
* This prevents us from cheating by thinking into opponent's time for the last AuxEngine search.
* The cheating was only minor, on my machine ~20 AuxEngine evals are done per move in ~70ms each. So AuxEngine would think an extra 35ms per move on average.
* Fix off by one error in AuxEngineFollowPvDepth.
* Setting this to 1 is supposed to only update the first move, but it would update 2 moves.
* Also update default from 3 to 4 so the behavior is the same as previous releases.
* Add some more logging stats

Code: Select all

auxengine_queue_ size 252 Average duration 72ms Number of evals 23 Number of updates 64
...just a very big +1 for doing this KillerDucky - it's a real contribution to the computer chess community! Can you tell us more on how you use it , what settings you use and why? Still trying to get my head wrapped around this - I want to use it mostly for analysis - so any suggestions or comments you might have would be useful. Thanks in advance.
Image

pohl4711
Posts: 1269
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by pohl4711 » Tue Feb 26, 2019 10:07 am

KillerDucky wrote:
Tue Feb 26, 2019 2:25 am
@pohl4711: Thanks for the logfile, I was able to spot the problem with it! Sorry I didn't test this myself. :oops: It happens with v0.4 if you do not use Syzygy. Fixed in v0.5!

https://github.com/killerducky/lc0/rele ... -fish-v0.5

* Fix crash in v0.4 when Syzygy is not used.
* Stop the AuxEngine when lc0 is done searching.
* This prevents us from cheating by thinking into opponent's time for the last AuxEngine search.
* The cheating was only minor, on my machine ~20 AuxEngine evals are done per move in ~70ms each. So AuxEngine would think an extra 35ms per move on average.
* Fix off by one error in AuxEngineFollowPvDepth.
* Setting this to 1 is supposed to only update the first move, but it would update 2 moves.
* Also update default from 3 to 4 so the behavior is the same as previous releases.
* Add some more logging stats

Code: Select all

auxengine_queue_ size 252 Average duration 72ms Number of evals 23 Number of updates 64
I still strongly recommend to use go movestime for the aux-engine, not a fixed depth, this should lead to higher depths in endgame and more Elo-gain at this weak point of Leela !

Jesse Gersenson
Posts: 582
Joined: Sat Aug 20, 2011 7:43 am
Contact:

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by Jesse Gersenson » Tue Feb 26, 2019 10:08 am

If I have a machine with 72 threads, how many would I assign to the auxengine?

pohl4711
Posts: 1269
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by pohl4711 » Tue Feb 26, 2019 11:50 am

An early intermediate result of LeelaFish in my long thinking-time tournament (86 games only) looks very promising...
Stockfish 10 vs. LeelaFish v0.3 N:32930 (the TCEC Superfinal 14 Net) and AuxEngine: Houdini 6 (Depth = 16, Threshold = 100)

https://www.sp-cc.de/long-thinkingtime.htm

User avatar
sincerelybrent
Posts: 47
Joined: Fri Feb 15, 2019 10:41 pm
Full name: Brent M

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by sincerelybrent » Tue Feb 26, 2019 11:54 am

0.5 is looking good, KillerDucky, but game 12 threw an exception for me. Here is the log:
http://www.westportchessclub.org/public ... ception.7z

Otherwise, with a LeelaRatio of 1.0, LeelaFish 0.5 w/ Houdini 6 (1 thread, 1024 hash, tactical mode=1) is beating Stockfish 10 64-bit POPCNT by 4 games in 50 total.

Javier Ros
Posts: 184
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by Javier Ros » Tue Feb 26, 2019 6:05 pm

Many thanks to KillerDucky for this great idea!
It works perfectly for me.

I wonder if it is possible to specify Houdini Tactical Mode adding in Arena:

Hash=1024;Threads=1;Tactical Mode=1

in AuxEngineOptions
The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.

dkappe
Posts: 385
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by dkappe » Tue Feb 26, 2019 6:36 pm

Trying out 0.5. I’ve noticed that with lots of aux engines (cheng4, sting 14 to name a few) it gives me a segmentation fault. Winter is an engine that works perfectly. I suspect that the info string parsing is a bit brittle.

KillerDucky
Posts: 12
Joined: Fri Mar 23, 2018 1:31 am

Re: Trying out the Leela Hybrid engine (NN with AB)

Post by KillerDucky » Wed Feb 27, 2019 12:45 am

I have a fix for one crash. Here is a test version:
https://ci.appveyor.com/project/killerd ... s/22675046
(Click on cuda/opencl/blas, then click on Aritifacts, then download lc0.exe).

@sincerelybrent I don't think this will fix your case, it looks different. I added a little more logging that might help narrow it down.

@pohl4711 yes I think using movetime instead of depth is better. I will add that after fixing some of these crashes.

Post Reply