In case you missed it: SSDF, shiny new hardware...

[Dann Corbit]

It is true that if you knew the opening book used by your opponent you could spend a lot of time and make a counter book to help your winning chances. Hard, but with enough resources it could be done. Komodo has just used a general opening book tuned to lines it understands well, and not against a specific opponent. It is not very practical to have a unique book for each opponent. We do change some lines to prevent repeating games, and Erdo uses his bets judgement to select the first few moves.

I agree it would be unfair to allow one program to use an opening book and another one not. But just as adding an evaluation or search modification to a program can make that program stronger, a well tuned book can help a program play stronger. But I cannot call me adding a new evaluation component to Komodo "unfair" anymore than someone writing a better book.

As long as both parties are allowed to have a book they choose, how is it unfair?

I personally like testing without opening books, but only because I am lazy and am not skilled at making such books. But if you show up to an event like WCCC without a book, your chances will drop a lot.

No you don't tune your book against all opponents just against the strongest one, or in best case strongest two. That seems to be what Junior did on WCCC. Tuned its book against Komodo.

On VLTC and strong hardware the only impact of a general book (tuned for engine, not against particular one) is to save time where it matters the most - in the opening, so with deep enough book you can save a lot of time that converts directly to Elo.

I agree books biggest benefits are to save time, but also to prevent the engine from going down some line too deep for it to find it is in trouble. When making Komodo books our goal is to get out of book early in a position Komodo likes. Long book lines tend to be drawish.

We do not tune against any program except Komodo. Testing using an unrelated engine takes more games to determine if a specific change helps or hurts. Every few months we might do a run against Stockfish, but it is pretty rare. More of a regression run versus another program. I think Larry did a run against Houdini this year, but that is even more rare. 99.9x% of our runs are between Komodo versions. I am not sure the value of the x part, but runs against other programs are very rare. Stockfish also seems to run against Stockfish to determine if a change helps.

If you play blitz, then every sound gambit needs a book line.
Engines will make bad decisions on sound gambits in short time control.

Also, for highly closed positions, book moves are a very good idea.

If you have TCEC type hardware and play at correspondence pace, then a book is not necessary unless you like variety.

[matejst]

I rememebr many years ago, when SSDF was about the only existing rating list, they included all top engines and they really did a great job.

but that was once.

times have changed now and they are doing a really poor job currently.

LTC is an excellent option, but having books with no specified depth(at least specify some depth limit), and no clear entry conditions is what makes the list almost copletely irrelevant.

entries are most important, of course, with other issues of less salient nature.

what is SSDF policy on entries?

I didn't feel that they ever did a great job -- it was debated a lot years ago and Milos reminded us of a key problem in one of his posts -- an aspect of their methodology was and remains subject to controversy, especially in years when their lists affected directly the sales of commercial programs. But I don't think it's the question of opening books: when one buys a complete package, a chess program, he's untitled to get also a good opening book. Many of us still use engines to analyse and prepare for OTB games.

On another side, the SSDF policy on entries, although implicit, is not so difficult to grasp, and there's nothing we can do about. I guess there are multiple aspects to this, some I agree with, and some I don't quite understand, but that's what it is.

But anyway, I am glad they started testing Wasp, and continued testing ProDeo, two engines I like very much.

[Dann Corbit]

I rememebr many years ago, when SSDF was about the only existing rating list, they included all top engines and they really did a great job.

but that was once.

times have changed now and they are doing a really poor job currently.

LTC is an excellent option, but having books with no specified depth(at least specify some depth limit), and no clear entry conditions is what makes the list almost copletely irrelevant.

entries are most important, of course, with other issues of less salient nature.

what is SSDF policy on entries?

I didn't feel that they ever did a great job -- it was debated a lot years ago and Milos reminded us of a key problem in one of his posts -- an aspect of their methodology was and remains subject to controversy, especially in years when their lists affected directly the sales of commercial programs. But I don't think it's the question of opening books: when one buys a complete package, a chess program, he's untitled to get also a good opening book. Many of us still use engines to analyse and prepare for OTB games.

On another side, the SSDF policy on entries, although implicit, is not so difficult to grasp, and there's nothing we can do about. I guess there are multiple aspects to this, some I agree with, and some I don't quite understand, but that's what it is.

But anyway, I am glad they started testing Wasp, and continued testing ProDeo, two engines I like very much.

Here is what they are testing:
The strength of the software system that you purchased on equivalent hardare.

So you purchased a program, you can use it.
It came with a book, you can use it.
It came with tablebase access, you can use it.

It is exactly the strength you would measure out of the box on your own home system if you used it as it comes on the same hardware that they are using.

It measures nothing more and nothing less.

I agree that part of the strength measured will be a function of the book in place.
I agree that part of the strength measured will be a function of the tablebase access.
But that is what you purchase when you purchase the product.

Yes, this is different than what other people test.
But quite frankly, no two testing groups test using identical conditions. And that is good.

If you are going to play chess online with your computer, the number that you really want to know for Elo is the SSDF version, unless you intend to delete the book and remove the tablebase files.

If you want to analyze chess positions with the book turned off (like I do) then the CCRL and CEGT numbers are measuring what you are interested in.

I guess that different people have interest in different things.

[Lyudmil Tsvetkov]

I rememebr many years ago, when SSDF was about the only existing rating list, they included all top engines and they really did a great job.

but that was once.

times have changed now and they are doing a really poor job currently.

LTC is an excellent option, but having books with no specified depth(at least specify some depth limit), and no clear entry conditions is what makes the list almost copletely irrelevant.

entries are most important, of course, with other issues of less salient nature.

what is SSDF policy on entries?

I didn't feel that they ever did a great job -- it was debated a lot years ago and Milos reminded us of a key problem in one of his posts -- an aspect of their methodology was and remains subject to controversy, especially in years when their lists affected directly the sales of commercial programs. But I don't think it's the question of opening books: when one buys a complete package, a chess program, he's untitled to get also a good opening book. Many of us still use engines to analyse and prepare for OTB games.

On another side, the SSDF policy on entries, although implicit, is not so difficult to grasp, and there's nothing we can do about. I guess there are multiple aspects to this, some I agree with, and some I don't quite understand, but that's what it is.

But anyway, I am glad they started testing Wasp, and continued testing ProDeo, two engines I like very much.

and I am happy about the Mephisto Gideon(or was it Escalibur?)

[Lyudmil Tsvetkov]

It is true that if you knew the opening book used by your opponent you could spend a lot of time and make a counter book to help your winning chances. Hard, but with enough resources it could be done. Komodo has just used a general opening book tuned to lines it understands well, and not against a specific opponent. It is not very practical to have a unique book for each opponent. We do change some lines to prevent repeating games, and Erdo uses his bets judgement to select the first few moves.

I agree it would be unfair to allow one program to use an opening book and another one not. But just as adding an evaluation or search modification to a program can make that program stronger, a well tuned book can help a program play stronger. But I cannot call me adding a new evaluation component to Komodo "unfair" anymore than someone writing a better book.

As long as both parties are allowed to have a book they choose, how is it unfair?

I personally like testing without opening books, but only because I am lazy and am not skilled at making such books. But if you show up to an event like WCCC without a book, your chances will drop a lot.

No you don't tune your book against all opponents just against the strongest one, or in best case strongest two. That seems to be what Junior did on WCCC. Tuned its book against Komodo.

On VLTC and strong hardware the only impact of a general book (tuned for engine, not against particular one) is to save time where it matters the most - in the opening, so with deep enough book you can save a lot of time that converts directly to Elo.

I agree books biggest benefits are to save time, but also to prevent the engine from going down some line too deep for it to find it is in trouble. When making Komodo books our goal is to get out of book early in a position Komodo likes. Long book lines tend to be drawish.

We do not tune against any program except Komodo. Testing using an unrelated engine takes more games to determine if a specific change helps or hurts. Every few months we might do a run against Stockfish, but it is pretty rare. More of a regression run versus another program. I think Larry did a run against Houdini this year, but that is even more rare. 99.9x% of our runs are between Komodo versions. I am not sure the value of the x part, but runs against other programs are very rare. Stockfish also seems to run against Stockfish to determine if a change helps.

If you play blitz, then every sound gambit needs a book line.
Engines will make bad decisions on sound gambits in short time control.

Also, for highly closed positions, book moves are a very good idea.

If you have TCEC type hardware and play at correspondence pace, then a book is not necessary unless you like variety.

if we want to teach our engines to play better chess, books are unnecessary.

[Dann Corbit]

It is true that if you knew the opening book used by your opponent you could spend a lot of time and make a counter book to help your winning chances. Hard, but with enough resources it could be done. Komodo has just used a general opening book tuned to lines it understands well, and not against a specific opponent. It is not very practical to have a unique book for each opponent. We do change some lines to prevent repeating games, and Erdo uses his bets judgement to select the first few moves.

I agree it would be unfair to allow one program to use an opening book and another one not. But just as adding an evaluation or search modification to a program can make that program stronger, a well tuned book can help a program play stronger. But I cannot call me adding a new evaluation component to Komodo "unfair" anymore than someone writing a better book.

As long as both parties are allowed to have a book they choose, how is it unfair?

I personally like testing without opening books, but only because I am lazy and am not skilled at making such books. But if you show up to an event like WCCC without a book, your chances will drop a lot.

No you don't tune your book against all opponents just against the strongest one, or in best case strongest two. That seems to be what Junior did on WCCC. Tuned its book against Komodo.

On VLTC and strong hardware the only impact of a general book (tuned for engine, not against particular one) is to save time where it matters the most - in the opening, so with deep enough book you can save a lot of time that converts directly to Elo.

I agree books biggest benefits are to save time, but also to prevent the engine from going down some line too deep for it to find it is in trouble. When making Komodo books our goal is to get out of book early in a position Komodo likes. Long book lines tend to be drawish.

We do not tune against any program except Komodo. Testing using an unrelated engine takes more games to determine if a specific change helps or hurts. Every few months we might do a run against Stockfish, but it is pretty rare. More of a regression run versus another program. I think Larry did a run against Houdini this year, but that is even more rare. 99.9x% of our runs are between Komodo versions. I am not sure the value of the x part, but runs against other programs are very rare. Stockfish also seems to run against Stockfish to determine if a change helps.

If you play blitz, then every sound gambit needs a book line.
Engines will make bad decisions on sound gambits in short time control.

Also, for highly closed positions, book moves are a very good idea.

If you have TCEC type hardware and play at correspondence pace, then a book is not necessary unless you like variety.

if we want to teach our engines to play better chess, books are unnecessary.

It is only necessary if you want to win more games.

[corres]

The one of the main issue with SSDF is its Elo list based on games played different hardware: beginning with old chess machine and continuing Intel 486 33 MHz, Intel Pentium 90 MHz, Intel Pentium 200 MHz, AMD K6-2 450 MHz, AMD Athlon 1200 MHz, Intel Core 2 Quad 2400MHz and now AMD Ryzen 1800x 3600 MHz. This is like as a competition with participant from - let's speak - Ford T-Model to Formula 1 super cars.

[mclane]

That's called competition.
It's normal that over the years you use different hardware.
When I am testing Komodo or other engines I do also use faster hardware then I used years ago.

Even didicated chess computers have today faster hardware.
Best example is Millennium chess exclusive, running chess genius on 300 MHz instead of 12 in the old days.

[matejst]

and I am happy about the Mephisto Gideon(or was it Escalibur?)

I am glad you like this program.

Joking aside, sometimes one tries to find an engine with a humanlike style, a chess program whose moves he is able to understand, and that's what I get with Wasp and ProDeo, Zappa or some earlier versions of Komodo.

And I feel that this obsession with engine "strength" is not sane, and that the quality of engines deteriorated in a way.

[corres]

[quote="mclane"]

That's called competition.

[/quote]

But what kind of competition?
Every good chess program are optimized (more or less) to the hardware used that time. You want a competition between chess programs or between hardware?
Moreover your lists were more founded if you would play chess programs
against each other directly: (for e.g.) Komodo running on Ryzen 1800x and Mephisto Lyon 68020 12 MHz (this is my chess machine).
Without direct matches the value of your lists are rather dubious.