Stockfish based (ThothFishv1.0) Leela-Killer

[Laskos]

ThothFishv1.0 described here is an excellent Stockfish modification:
viewtopic.php?f=2&t=70300

You can read there how it can be configured and it is a great Stockfish mod. As I knew that Leela is very strong while many pieces are on the board, and the games are "regular", i.e. start with regular, human-like openings, I used ThothFish UCI menu options from "pawn exchange" to "queen exchange" as positive +50 (all of them). That was found as performing well after several other trials against Leela. These settings favor quick exchange of pieces, shortening the openings and early midgame, where Leela is very strong, and insist more on late midgame and endgame, where SF is stronger.
ThothFish is on 4 strong i7 cores, Lc0 v21.1 on RTX 2070 uses the net ID41687, one of the strongest I checked (almost surely stronger than all T30 nets). Openings were 8-pliers of gm2600.pgn file, GM short openings which Leela likes quite a bit. Having on my PC a high Leela Ratio of 2.0-2.5, the results of Lc0 against regular SF_dev are very good, Leela beats consistently SF_dev:

Games at 30s + 0.5s:

Score of lc0_41687 vs SF_dev: 34 - 14 - 52 [0.600] 100
Elo difference: 70.44 +/- 47.31
Finished match

Score of lc0_41687 vs SF_dev: 37 - 15 - 48 [0.610] 100
Elo difference: 77.71 +/- 49.45
Finished match

Both matches show a heavy Leela dominance over SF_dev.


But against ThothFish with above mentioned settings favoring quick exchanges and emphasizing the endgames, I got from the same GM short openings:

Score of lc0_41687 vs SF_Thoth: 20 - 18 - 62 [0.510] 100
Elo difference: 6.95 +/- 42.12
Finished match

Score of lc0_41687 vs SF_Thoth: 22 - 20 - 58 [0.510] 100
Elo difference: 6.95 +/- 44.30
Finished match

You have to keep in mind that my Leela Ratio is 2-2.5 (strong GPU), but in TCEC conditions (Leela Ratio of about 1), it's very probable ThothFish would have beaten Leela, maybe even conclusively. ThothFish gained 65 Elo points (well, error of +/- 40 Elo points 2SD) over Leela compared to SF_dev. It's maybe time for SF devs to think of anti-Leela mods of SF if, for example, SF meets Leela in the TCEC superfinal. Things look like the next TCEC superfinal, if hardware remains the same, will go in Leela T40 favor, if Stockfish team will send the regular SF_dev.

One thing to note: it's easier for now to build an anti-Leela SF than an anti-SF Leela. Regular AB engines are more generalistic and adaptable when confronted with unfamiliar positions and the same goes with manually adapting them to Leela's known strengths and weaknesses. The level of abstraction of manual fiddling with eval of regular engines is higher than the level of abstraction of NN based engines, which are more a sort of "specialists".

[Uri Blass]

I wonder how many rating points ThothFish1.0 leela-killer is losing against other engines or what is the result of it against stockfish that it is derived from it.

[Laskos]

ThothFishv1.0 described here is an excellent Stockfish modification:
viewtopic.php?f=2&t=70300

You can read there how it can be configured and it is a great Stockfish mod. As I knew that Leela is very strong while many pieces are on the board, and the games are "regular", i.e. start with regular, human-like openings, I used ThothFish UCI menu options from "pawn exchange" to "queen exchange" as positive +50 (all of them). That was found as performing well after several other trials against Leela. These settings favor quick exchange of pieces, shortening the openings and early midgame, where Leela is very strong, and insist more on late midgame and endgame, where SF is stronger.
ThothFish is on 4 strong i7 cores, Lc0 v21.1 on RTX 2070 uses the net ID41687, one of the strongest I checked (almost surely stronger than all T30 nets). Openings were 8-pliers of gm2600.pgn file, GM short openings which Leela likes quite a bit. Having on my PC a high Leela Ratio of 2.0-2.5, the results of Lc0 against regular SF_dev are very good, Leela beats consistently SF_dev:

Games at 30s + 0.5s:

Score of lc0_41687 vs SF_dev: 34 - 14 - 52 [0.600] 100
Elo difference: 70.44 +/- 47.31
Finished match

Score of lc0_41687 vs SF_dev: 37 - 15 - 48 [0.610] 100
Elo difference: 77.71 +/- 49.45
Finished match

Both matches show a heavy Leela dominance over SF_dev.


But against ThothFish with above mentioned settings favoring quick exchanges and emphasizing the endgames, I got from the same GM short openings:

Score of lc0_41687 vs SF_Thoth: 20 - 18 - 62 [0.510] 100
Elo difference: 6.95 +/- 42.12
Finished match

Score of lc0_41687 vs SF_Thoth: 22 - 20 - 58 [0.510] 100
Elo difference: 6.95 +/- 44.30
Finished match

You have to keep in mind that my Leela Ratio is 2-2.5 (strong GPU), but in TCEC conditions (Leela Ratio of about 1), it's very probable ThothFish would have beaten Leela, maybe even conclusively. ThothFish gained 65 Elo points (well, error of +/- 40 Elo points 2SD) over Leela compared to SF_dev. It's maybe time for SF devs to think of anti-Leela mods of SF if, for example, SF meets Leela in the TCEC superfinal. Things look like the next TCEC superfinal, if hardware remains the same, will go in Leela T40 favor, if Stockfish team will send the regular SF_dev.

One thing to note: it's easier for now to build an anti-Leela SF than an anti-SF Leela. Regular AB engines are more generalistic and adaptable when confronted with unfamiliar positions and the same goes with manually adapting them to Leela's known strengths and weaknesses. The level of abstraction of manual fiddling with eval of regular engines is higher than the level of abstraction of NN based engines, which are more a sort of "specialists".

Still early to say, but the ThothFish mod as used against Leela seems to lose only some maybe 25 Elo points against regular SF_dev in direct match against it (still have to finish the run, too early). It probably means that ThothFish mod can be used to start with in later TCEC stages, especially if there are several NN based engines (where the mod performs better than SF_dev). It must be emphasized that these gains against Leela are against a Zero T40 network, if one starts fiddling manually with Leela nets, for example combining T40 and T40 "Ender" or T40 AntiSF, maybe the gains will look different. But again, it's easier to adapt a transparent regular eval of AB engines to known strengths and weaknesses of Leela, than for a NN based engine to adapt to particular conditions not harming other conditions.

[Laskos]

I wonder how many rating points ThothFish1.0 leela-killer is losing against other engines or what is the result of it against stockfish that it is derived from it.

Just running this now. We posted almost simultaneously, read my previous post.

[Laskos]

I wonder how many rating points ThothFish1.0 leela-killer is losing against other engines or what is the result of it against stockfish that it is derived from it.

Just running this now. We posted almost simultaneously, read my previous post.

Too small a regression to measure in 400 games at 30s + 0.5s:

Score of SF_dev vs SF_Thoth: 90 - 76 - 234 [0.517] 400
Elo difference: 12.17 +/- 21.93
Finished match

SF_dev is the original engine, modified in SF_Toth (both are very recent, 24th of March).
So, the weakening doesn't seem very relevant, and it seems ThothFish trades the pieces quite soundly and doesn't make much concessions in the process. Not sure how would they have comparatively performed against a weak engine.

[corres]

In the old time developers think the opened positions give more opportunity to utilize the better tactical capacity of (AB type) engines.
Maybe reacting to the effect of NN engines the developers of AB engines should think such a manner than it was in the past.
But I think for a long run the competition will be won by the NN engines.

[Raphexon]

I guess this confirms my suspicion that aggressively trading against NN engines (by an AB engine) is advantageous to the AB engine.

Basically you have the positional stage when there are still a lot of pieces on the board and the tactical stage when there are few.
And by shortening the positional stage through aggressive exchanges, the NN-engine is given less time to estabilish a positional advantage.

Great work!

[Leo]

Very interesting.

[jp]

ThothFish gained 65 Elo points (well, error of +/- 40 Elo points 2SD) over Leela compared to SF_dev. It's maybe time for SF devs to think of anti-Leela mods of SF if, for example, SF meets Leela in the TCEC superfinal. Things look like the next TCEC superfinal, if hardware remains the same, will go in Leela T40 favor, if Stockfish team will send the regular SF_dev.

Does it still have this advantage if Leela has 6-man TBs, which it does in TCEC?

[Laskos]

ThothFish gained 65 Elo points (well, error of +/- 40 Elo points 2SD) over Leela compared to SF_dev. It's maybe time for SF devs to think of anti-Leela mods of SF if, for example, SF meets Leela in the TCEC superfinal. Things look like the next TCEC superfinal, if hardware remains the same, will go in Leela T40 favor, if Stockfish team will send the regular SF_dev.

Does it still have this advantage if Leela has 6-man TBs, which it does in TCEC?

Haven't checked, but in the past I did check Leela performance with 6-men Syzygy on tricky endgames, they helped, but seemed to help no more than TBs help the regular engines, although strong regular engines play much better endgames even without TBs. It seems there are much more TB cut-offs with regular engines, and they actually use TBs better than Leela does. So, I wouldn't expect that using TBs something changes, and the games, though having more piece trading, are not going close to TBs very quickly, the importance of TBs being low even here.