SF13 vs FF2

[gaard]

Threads=1
Contempt=0
Analysis Contempt=Off
Hash=128
TC=60"+1"

Positions: swapped
Openings: 2moves_v2.pgn
syzygy: none
CPU: i7-9750H

cutechess-cli: -resign movecount=5 score=900

ordo: -W -D

compilation: make build ARCH=x86-64-bmi2 COMP=mingw

Code: Select all

   # PLAYER                : RATING    POINTS  PLAYED    (%)
   1 Stockfish 13          :    0.0     973.5    1691   57.6%
   2 Fat Fritz 2 Private   :  -24.5     887.0    1691   52.5%
   3 Fat Fritz 2 Public    :  -84.1     676.5    1692   40.0%

White advantage = 33.30
Draw rate (equal opponents) = 80.26 %

2537 of 9600 games completed:

More to follow...

[gaard]

I decided to end the tournament at 3200 games.

Code: Select all

   # PLAYER                 : RATING    POINTS  PLAYED    (%)
   1 Stockfish 13           :    0.0    1233.5    2132   57.9%
   2 Fat Fritz 2 Private    :  -27.5    1112.0    2134   52.1%
   3 Fat Fritz 2 Public     :  -85.2     854.5    2134   40.0%

White advantage = 31.27
Draw rate (equal opponents) = 80.05 %

Games:

[Modern Times]

The vast majority of tests show FF2 losing head-to-head vs SF13. Stefan Pohl's is the only one I've seen where FF2 wins, by a small margin.

I ran some tests of my own.

First One

All 960 FRC positions, played reversed sides, so 1,920 games
TimeControl 120+1
256MB hash
1 thread
5-men sysgy

Score of Stockfish 13 vs Fat Fritz 2 : 201 - 168 - 1551 [0.509]
Elo difference: 6.0 +/- 6.8, LOS: 95.7 %, DrawRatio: 80.8 %

Second One

Pohl's 5mvs_balanced pgn (2,871 positions, reversed sides so 5,742 games)
(not sure how balanced this is in terms of ECO mix)
TimeControl 90+1
256MB hash
1 thread
5-men sysgy

1 Stockfish 13 +11 +406/=5118/-218 51.64% 2965.0/5742
2 Fat Fritz 2 -11 +218/=5118/-406 48.36% 2777.0/5742

All too close to call and within error margins. But the question is does the FF2 NNUE offer anything special for analysis purposes ? That is the important thing for most people. I can't answer that. Engine Elo is a guide but not always definitive.

[carldaman]

I've only got the free FF2 net, and it plays a more aggressive and imbalanced game than vanilla SF NNUE, especially with Black.

I find it interesting, but I also think 100 euro is a little too steep a price right now. Maybe I'll wait for their bi-annual sale and get it cheaper that way.

[gaard]

The vast majority of tests show FF2 losing head-to-head vs SF13. Stefan Pohl's is the only one I've seen where FF2 wins, by a small margin.

I ran some tests of my own.

First One

All 960 FRC positions, played reversed sides, so 1,920 games
TimeControl 120+1
256MB hash
1 thread
5-men sysgy

Score of Stockfish 13 vs Fat Fritz 2 : 201 - 168 - 1551 [0.509]
Elo difference: 6.0 +/- 6.8, LOS: 95.7 %, DrawRatio: 80.8 %

Second One

Pohl's 5mvs_balanced pgn (2,871 positions, reversed sides so 5,742 games)
(not sure how balanced this is in terms of ECO mix)
TimeControl 90+1
256MB hash
1 thread
5-men sysgy

1 Stockfish 13 +11 +406/=5118/-218 51.64% 2965.0/5742
2 Fat Fritz 2 -11 +218/=5118/-406 48.36% 2777.0/5742

All too close to call and within error margins. But the question is does the FF2 NNUE offer anything special for analysis purposes ? That is the important thing for most people. I can't answer that. Engine Elo is a guide but not always definitive.

AS recommended more representative openings, which would definitely shrink these differences, and is perhaps more "fair". However, your first results show a LOS of 95.7%, and your next results a LOS of 99.99%+, which is to say, the error margins are negligible. Correct me if I'm wrong.

[gaard]

I've only got the free FF2 net, and it plays a more aggressive and imbalanced game than vanilla SF NNUE, especially with Black.

I find it interesting, but I also think 100 euro is a little too steep a price right now. Maybe I'll wait for their bi-annual sale and get it cheaper that way.

I'm not sure the "private" net wasn't meant to be made public, and there is actually a link to it on the second page of CCC: GT.