STS rating v13.1 for Lc0 0.21.2 with nodes = 1

[Guenther]

STS is not great positional test suite and this became clear precisely with Leela. I have my own 3 year old positional opening suite containing 200 positions
...

Is this test still available for download somewhere?
My forum research only found a dead link in a post sixteen months ago (not three years?).

viewtopic.php?f=2&t=66535&p=750162&hili ... st#p750203

[Laskos]

I started talking to you because I disagreed about things you said about STS and he was right. I don't think 200 positions can ever be proof of overall superiority. I did Kai's test and 2 program in the range of 2400-2500 elo are in the middle between the tops.

Code: Select all

Engine: Lc0 v0.21.2-rc1         125/200
Engine: Stockfish 10             93/200
Engine: Ethereal 11.25           86/200
Engine: Senpai 1.0               78/200
Engine: Mephisto Gideon          76/200
Engine: Xiphos 0.5               72/200
Engine: Laser 1.6                71/200
Engine: Rebel Century            66/200
Engine: Sting SF 9.6             66/200
Engine: Texel 1.06a45            65/200
Engine: Rodent III               64/200
Engine: Rybka 4.1                61/200

Thanks. This is _positional_ test suite, not "strength" test suite. You results look reasonable (on strong GPU you will get even higher results for Leela late 20b nets). Mephisto Gideon can well be positionally above some recent much stronger engines. There is emphasis nowadays on a standard by now very efficient search.

My suite is rough and unpolished. I guess 10% of my solutions are wrong, and another 10% dubious. But seeing Lc0 at longer times on strong GPU solving about 80%, I get confidence in both: my suite is not totally useless AND in that Lc0 in the openings is a positional phenomenon. I guess the opening theory will soon be affected by NN-based engines.

[Laskos]

STS is not great positional test suite and this became clear precisely with Leela. I have my own 3 year old positional opening suite containing 200 positions
...

Is this test still available for download somewhere?
My forum research only found a dead link in a post sixteen months ago (not three years?).

viewtopic.php?f=2&t=66535&p=750162&hili ... st#p750203

I am on phone now, try this:
viewtopic.php?f=2&t=70438&p=795360&hili ... 00#p795360

I think I bundled together in a zip file the opening suite with the even more dubious midgame positional suite (I have an updated version of the latter).

[Rebel]

But to me the only reason for a test- suite to get outdated was, if it was solved too easily and too completely by new arising engines. No reason to update because a single one new engine doesn't perform good enough for some fans of the new engine.

That's pretty unreasonable to Dann & Swami considering the energy and massive computer time they spend. It was good at the time, now it needs an update.
...
I don't believe that happened with the creation of STS. In the 90's people cooperated online to create a tactical suite called ECM. Many programmers profited despite its errors.

I didn't mean STS being outdated neither bulit for a single engine (neither Rybka nor any other of that time) I meant Kai's one.

But as I said before I wanted to talk about STS, take it up with Kai.

[Rebel]

My suite is rough and unpolished. I guess 10% of my solutions are wrong, and another 10% dubious. But seeing Lc0 at longer times on strong GPU solving about 80%, I get confidence in both: my suite is not totally useless AND in that Lc0 in the openings is a positional phenomenon. I guess the opening theory will soon be affected by NN-based engines.

The red is absolutely true. I was studying the one ply result of Lc0 (see the Whatever is current - Amazing Leela thread) and it amazes me that just on one ply it created the "Ruy Lopez - closed" main variation all by its own.

1.e4 e5 2.Nf3 Nc6 3.Bb5 a6 4.Ba4 Nf6 5.O-O Be7 6.Re1 b5 7.Bb3 d6 8.c3 O-O 9.h3 Bb7 10.d4 Re8 11.Nbd2 Bf8 12.Bc2 h6 13.Nf1

[chrisw]

My suite is rough and unpolished. I guess 10% of my solutions are wrong, and another 10% dubious. But seeing Lc0 at longer times on strong GPU solving about 80%, I get confidence in both: my suite is not totally useless AND in that Lc0 in the openings is a positional phenomenon. I guess the opening theory will soon be affected by NN-based engines.

The red is absolutely true. I was studying the one ply result of Lc0 (see the Whatever is current - Amazing Leela thread) and it amazes me that just on one ply it created the "Ruy Lopez - closed" main variation all by its own.

1.e4 e5 2.Nf3 Nc6 3.Bb5 a6 4.Ba4 Nf6 5.O-O Be7 6.Re1 b5 7.Bb3 d6 8.c3 O-O 9.h3 Bb7 10.d4 Re8 11.Nbd2 Bf8 12.Bc2 h6 13.Nf1

The closer you get to the root of the game, the more the NN is a database lookup of the statistics of the input games, and since the input games are self-generated at something beyond 3000 Elo, it ought to be able to steer to various unknown as yet traps and re-assessments of known and unknown lines.

[Rebel]

My suite is rough and unpolished.

Question about your suite, how long does it take (an estimation is also ok) to analyze your 200 positions at (say) depth=5 with a 256x20 NN on your (GPU) hardware.

[Laskos]

My suite is rough and unpolished.

Question about your suite, how long does it take (an estimation is also ok) to analyze your 200 positions at (say) depth=5 with a 256x20 NN on your (GPU) hardware.

Not sure about depth, I am testing at fixed time, but depth=5 is really low for current Lc0 on my GPU. The times are from 1s per position (about 15,000 nodes) to 120s per position (about 2,500,000 nodes).

[Rebel]

My suite is rough and unpolished.

Question about your suite, how long does it take (an estimation is also ok) to analyze your 200 positions at (say) depth=5 with a 256x20 NN on your (GPU) hardware.

Not sure about depth, I am testing at fixed time, but depth=5 is really low for current Lc0 on my GPU. The times are from 1s per position (about 15,000 nodes) to 120s per position (about 2,500,000 nodes).

Thanks Kai. That's more than 100 x faster I have right now. I think I am going to buy some flowers for my wife tomorrow.

[Laskos]

My suite is rough and unpolished.

Question about your suite, how long does it take (an estimation is also ok) to analyze your 200 positions at (say) depth=5 with a 256x20 NN on your (GPU) hardware.

Not sure about depth, I am testing at fixed time, but depth=5 is really low for current Lc0 on my GPU. The times are from 1s per position (about 15,000 nodes) to 120s per position (about 2,500,000 nodes).

Thanks Kai. That's more than 100 x faster I have right now. I think I am going to buy some flowers for my wife tomorrow.