SMP SF Formula

[Raphexon]

So what's the issue?

You're expecting linear gains even though chess is a draw and SF14 is already extremely with 1 thread?

[Werewolf]

So what's the issue?

You're expecting linear gains even though chess is a draw and SF14 is already extremely with 1 thread?

Of course not.

[Werewolf]

Does anyone know of any software which can do engine vs engine testing with different time controls for each engine?

I was hoping Cutechess (for its concurrency) but I don't think it can. Aquarium can but it's clunky and decides by colour rather than engine.

[Raphexon]

1.Cutechess-Cli can.

2.Cutechess also has a pull request on github for assymetric TC, you can merge it on your own fork and then compile it.

3.You can also download an updated cutechess from here https://rwbc-chess.de/download.htm

During tournament setup, right click the desired engine to setup the TC you want (for that engine). You can mix and match whatever TC you want.

[Uri Blass]

Does anyone know of any software which can do engine vs engine testing with different time controls for each engine?

I was hoping Cutechess (for its concurrency) but I don't think it can. Aquarium can but it's clunky and decides by colour rather than engine.

Yes
Banksia gui can do it

[Werewolf]

1.Cutechess-Cli can.

2.Cutechess also has a pull request on github for assymetric TC, you can merge it on your own fork and then compile it.

3.You can also download an updated cutechess from here https://rwbc-chess.de/download.htm

During tournament setup, right click the desired engine to setup the TC you want (for that engine). You can mix and match whatever TC you want.

Very helpful, thanks.

I'm trying to test two things:
a) SF vs SF with greatly differing threads. This test is complete and shows little gain on 60 threads vs 1 thread.
b) SF vs SF with greatly differing time.

If b) also shows little gain, we're just running out of head-room in chess due to elo engine gain and SF being very hard to beat. But if b) yields much better results than a) I think it means there's an SMP issue with these NNUE engines.

[MikeB]

1.Cutechess-Cli can.

2.Cutechess also has a pull request on github for assymetric TC, you can merge it on your own fork and then compile it.

3.You can also download an updated cutechess from here https://rwbc-chess.de/download.htm

During tournament setup, right click the desired engine to setup the TC you want (for that engine). You can mix and match whatever TC you want.

Very helpful, thanks.

I'm trying to test two things:
a) SF vs SF with greatly differing threads. This test is complete and shows little gain on 60 threads vs 1 thread.
b) SF vs SF with greatly differing time.

If b) also shows little gain, we're just running out of head-room in chess due to elo engine gain and SF being very hard to beat. But if b) yields much better results than a) I think it means there's an SMP issue with these NNUE engines.

I am 90% confident it's the head room, the draw rates get so high, you might get slighty different results with different openings, I believe the SF tean just changed their openings again ( Ijust skimmed over some PRs), but you may want to grab those. Also depends on time controls - longer TC means more compressed Elo differences.

[smatovic]

Hmm, idk, but if the effective branching factor of current SF is still around 2 there must be something to gain via an parallel game tree search, not sure how the selective search heuristics play out in this.

--
Srdja

[Raphexon]

1.Cutechess-Cli can.

2.Cutechess also has a pull request on github for assymetric TC, you can merge it on your own fork and then compile it.

3.You can also download an updated cutechess from here https://rwbc-chess.de/download.htm

During tournament setup, right click the desired engine to setup the TC you want (for that engine). You can mix and match whatever TC you want.

Very helpful, thanks.

I'm trying to test two things:
a) SF vs SF with greatly differing threads. This test is complete and shows little gain on 60 threads vs 1 thread.
b) SF vs SF with greatly differing time.

If b) also shows little gain, we're just running out of head-room in chess due to elo engine gain and SF being very hard to beat. But if b) yields much better results than a) I think it means there's an SMP issue with these NNUE engines.

60x TC obviously beats 60x threads.
That has been the case with every SF.

Why would that mean SMP issues with NNUE?

[Werewolf]

1.Cutechess-Cli can.

2.Cutechess also has a pull request on github for assymetric TC, you can merge it on your own fork and then compile it.

3.You can also download an updated cutechess from here https://rwbc-chess.de/download.htm

During tournament setup, right click the desired engine to setup the TC you want (for that engine). You can mix and match whatever TC you want.

Very helpful, thanks.

I'm trying to test two things:
a) SF vs SF with greatly differing threads. This test is complete and shows little gain on 60 threads vs 1 thread.
b) SF vs SF with greatly differing time.

If b) also shows little gain, we're just running out of head-room in chess due to elo engine gain and SF being very hard to beat. But if b) yields much better results than a) I think it means there's an SMP issue with these NNUE engines.

60x TC obviously beats 60x threads.
That has been the case with every SF.

Why would that mean SMP issues with NNUE?

I wouldn't compare 60x more time with 60x more threads, as you say that would be an easy win for the 60x more time.

Instead what I did was try the match again by having 4x as much time.
So:

SF 14.1 at G10+10 vs SF 14.1 at G40+40
This time I used ponder OFF for obvious reasons.

Other conditions the same.

The result was:
65 draws
5 wins for the extra time version.

Obviously we're talking about a tiny sample size - so I can't say anything with any confidence - but it does seem to suggest 60x more threads is only slightly better than 4x as much time.

Personally I think it's a combination of a) and b) in my example above, but clearly more testing is needed.