In case you missed it: SSDF, shiny new hardware...

[zullil]

I am talking about permanent brain, that is the engine computes in opponents consideration time.

i.e., "pondering"

[Dann Corbit]

The SSDF is now using AMD Ryzen 7 1800X, which is an 8-core processor at 3.6 GHz. They have opted for 16 GB RAM as standard on this hardware.

Good hardware, bad software. Stockfish 6??

Because of the large number of games played, stockfish 6 is an excellent ruler.

Keep in mind that the SSDF plays at 40/moves in 2 hours. So it takes a lot of effort to test a program.

[carldaman]

I am not talking about permanent hash. I am talking about permanent brain, that is the engine computes in opponents consideration time.

When I play chess I do also have an opening book in mind and I do also compute when the opponents time is running.
It's a natural feature. So why should it be wrong to have an opening book.

The problem is that chess engines have a very precise memory that goes as deep and wide as one wishes, as far as opening books are concerned. This is very dissimilar to humans.

The effect of this is that you end up testing the distinct books as much or more than you're testing the engines, which is a huge negative for testing.
The only way I could allow own books is if they were capped at a low depth. For an engine rating list, it is ridiculous to have any engine reel off an indefinite number of moves straight from book, without any 'thinking' on their own.

CL

[Rebel]

tablebases provide equal conditions for all,

They are not, some have them, others don't.

That's an answer equal to "multi-core machines don't provide equal condition for all because some engines have SMP and some don't".
Pathetic excuses.

You are missing the context

No I am not, you are. Lyudmil is right here. There is a de-facto standard for EGBT - syzygy 6-men. Probing code and everything open-source. So any engines that doesn't have it implemented is to blame coz playing field is equal for everyone. With books it's a different game. Every engine uses it's own and even more ridiculous perfectly fine (and strong) engines without books are not allowed to play?!!!

Raw engine strength was the context, table bases don't belong to that.

[Rebel]

Moreover, regular books are just too weak already after move 10 for such a TC and strong machine. So you take Brainfish book, maybe tune it a bit, limit it to 10 moves and with the latest SF you can't loose a single game.

Maybe you should have a look at: http://rebel13.nl/prodeo/prodeo%202.1%20yat.html

I got an 102 ELO improvement. Maybe you underestimate the impact of a book with 150+ million positions?

Surely an 102 ELO improvement is not going to happen with SF8 (and others) but has your above idea (statement actually) been tried?

I fail to see what my comment has to do with yours?
Is your book 102 Elo stronger than Brainfish book trimmed to 10 moves?
First of all, I strongly doubt that Prodeo with your books would come out as stronger vs. Prodeo with Brainfish book trimmed to 10 moves.
For SF I'm pretty sure that Brainfish (with book trimmed to 10 moves) would destroy SF using only your book.

More statements without offered evidence.

[Henrik Dinesen]

The SSDF is now using AMD Ryzen 7 1800X, which is an 8-core processor at 3.6 GHz. They have opted for 16 GB RAM as standard on this hardware.

what is SSDF?

I understand in the past, but now?

We still honor them for what they did in the past, as first one offer a platform for competition long before we heard of Lyudmil Tsvetkov

We still honor them now for willing to play LTC games + permanent brain + opening books + allowing learning software, meaning a rating list where programs are allowed to compete on their strongest settings.

Exactly. SSDF is the very root of engine/chesscomputer testting. It's a huge work they do, and has done for so many years. I'd be surprised if the newer lists hasn't viewed SSDF when calibrating their own list, to some extend - though haven't read how precisely they're build. SSDF is the "massive" alternative - for me - to the weekly updated list.

[Lyudmil Tsvetkov]

The SSDF is now using AMD Ryzen 7 1800X, which is an 8-core processor at 3.6 GHz. They have opted for 16 GB RAM as standard on this hardware.

Good hardware, bad software. Stockfish 6??

Because of the large number of games played, stockfish 6 is an excellent ruler.

Keep in mind that the SSDF plays at 40/moves in 2 hours. So it takes a lot of effort to test a program.

no, I don't like the mouth of that gifted horse.

[Lyudmil Tsvetkov]

I haven't posted for years, although I am still reading regularly new threads, and remain very present on the forum. But I felt I had not much to say on many topics -- I regret though not to have thanked the many authors of engines and GUIs who offered us something different and original, useful for training and enjoying chess (Ed, Frank, Steven, HGM, Lucas, Brendan, Norman and many others).

After all these years, I still feel that the SSDF does an excellent job, and that their approach -- testing products, not engines -- herited from years when chess programs were complete software packages, not only engines, with a GUI, an opening book, is the right one. I like also the fact that they test at LTC -- it's probably the most important element for me.

I also do understand why they don't test Houdini, nor every version of engines. That said, in the comments of the chairman, I read that they soon will test the latest version of SF, so it's not that they avoid testing SF. Testing at LTC does severely limit the number of programs/engines they can test, so a choice has to be made. Some things could be done differently, but it's always easy to criticize.

For me, this list has still a lot to offer, especially since it's the only one testing dedicated chess computers, and conserving a testing paradigm I find is usefully complementing the work of other testing groups and individuals.

I rememebr many years ago, when SSDF was about the only existing rating list, they included all top engines and they really did a great job.

but that was once.

times have changed now and they are doing a really poor job currently.

LTC is an excellent option, but having books with no specified depth(at least specify some depth limit), and no clear entry conditions is what makes the list almost copletely irrelevant.

entries are most important, of course, with other issues of less salient nature.

what is SSDF policy on entries?

it is not possible not to test latest SF and Houdini, and test some 3-rd-rated engines instead.

[Lyudmil Tsvetkov]

The SSDF is now using AMD Ryzen 7 1800X, which is an 8-core processor at 3.6 GHz. They have opted for 16 GB RAM as standard on this hardware.

what is SSDF?

I understand in the past, but now?

We still honor them for what they did in the past, as first one offer a platform for competition long before we heard of Lyudmil Tsvetkov

We still honor them now for willing to play LTC games + permanent brain + opening books + allowing learning software, meaning a rating list where programs are allowed to compete on their strongest settings.

permanent brain = bad;
opening books = bad;
learning features = bad;

Are table bases also bad?

are you really that basic-level?

Nope, just pointing out your (expected) inconsistency.

tablebases provide equal conditions for all,

They are not, some have them, others don't.

opening books distort a fair competition.

Not at all, an opening book is an essential part of a chess program, in there the programmer decides what suits his program best.

CCRL | CEGT etc. are all fine, so is the SSDF concept.

how long an opening book could be?

any depth limits?

15 years ago, that was an excellent rating list, not now.

15 years ago, the list included Fritz, Junior, Gandalf, your engine, Crafty, etc., all the worthy competitors.

15 years ago, using books was meaningful, as engines played extremely weak in the opening and in general.

but not now, when we start having perfection-seeking engines.

a perfection-seeking engine does not need a book.

in any case, what most posters write here is inconsistent, as they defend one point of view(testing with custom books), while in reality, when testing their own engines, they unify the book or use no books at all.

it is difficult to discuss with people who state one thing, while doing the precise opposite.

[mjlef]

It is true that if you knew the opening book used by your opponent you could spend a lot of time and make a counter book to help your winning chances. Hard, but with enough resources it could be done. Komodo has just used a general opening book tuned to lines it understands well, and not against a specific opponent. It is not very practical to have a unique book for each opponent. We do change some lines to prevent repeating games, and Erdo uses his bets judgement to select the first few moves.

I agree it would be unfair to allow one program to use an opening book and another one not. But just as adding an evaluation or search modification to a program can make that program stronger, a well tuned book can help a program play stronger. But I cannot call me adding a new evaluation component to Komodo "unfair" anymore than someone writing a better book.

As long as both parties are allowed to have a book they choose, how is it unfair?

I personally like testing without opening books, but only because I am lazy and am not skilled at making such books. But if you show up to an event like WCCC without a book, your chances will drop a lot.

No you don't tune your book against all opponents just against the strongest one, or in best case strongest two. That seems to be what Junior did on WCCC. Tuned its book against Komodo.

On VLTC and strong hardware the only impact of a general book (tuned for engine, not against particular one) is to save time where it matters the most - in the opening, so with deep enough book you can save a lot of time that converts directly to Elo.

I agree books biggest benefits are to save time, but also to prevent the engine from going down some line too deep for it to find it is in trouble. When making Komodo books our goal is to get out of book early in a position Komodo likes. Long book lines tend to be drawish.

We do not tune against any program except Komodo. Testing using an unrelated engine takes more games to determine if a specific change helps or hurts. Every few months we might do a run against Stockfish, but it is pretty rare. More of a regression run versus another program. I think Larry did a run against Houdini this year, but that is even more rare. 99.9x% of our runs are between Komodo versions. I am not sure the value of the x part, but runs against other programs are very rare. Stockfish also seems to run against Stockfish to determine if a change helps.