Effect of "Own Book" vs Generic book

[Spock]

Following on a discussion in another thread - there seems to be quite a bit of interest in what an engine can gain by using it's own book, instead of the generic books used by all engines as in CEGT and CCRL

The only way to do this in my view is to do it in a scientific manner - exactly the same opponents and the same number of games, and of course the same machine with all other conditions identical.

Say we do Hiarcs, which has a reputation for a very good book

- Hiarcs generic book vs 10 opponents with generic book, 50 games each
- Hiarcs own book vs 10 opponents with generic book, 50 games each

This isolates the effect of Hiarcs using it's own book instead of a generic. And then run the games through ELOstat or bayeselo and compute the ratings. Whether 500 games each option is enough so that error margins are small enough is debateable.

Would have to be blitz of course.

Any other ideas ? I'm willing to run such a test

Question is: Should the generic book have any move restriction, of 12 moves say ? CEGT and CCRL have such a restriction, so that kind of makes sense, but then again in this little test the Hiarcs "own book" would have no such restriction, so it also makes sense to have the generic book unrestricted as well...

[Kirill Kryukov]

Hi Ray! Your experiment plan looks good. I think 500 games is not enough, I suggest at least 1000. For example using 20 opponents and 50 games per each. More opponents is always good, but those oponents should not be more than 200 points lower or higher than engine you test. IMO, some weaker engine would suit better because it will be easier to find enough opponents, both stronger and weaker than itself.

Question is: Should the generic book have any move restriction, of 12 moves say ? CEGT and CCRL have such a restriction, so that kind of makes sense, but then again in this little test the Hiarcs "own book" would have no such restriction, so it also makes sense to have the generic book unrestricted as well...

I'd say yes, book should have restriction. By testing with restriction we will be able to estimate the rating of Hiarcs (or other engine) with own book on CCRL scale (assuming all conditions are identical with one of CCRL lists). By testing without restricting own book, we will have no such estimation.

[bob]

Following on a discussion in another thread - there seems to be quite a bit of interest in what an engine can gain by using it's own book, instead of the generic books used by all engines as in CEGT and CCRL

The only way to do this in my view is to do it in a scientific manner - exactly the same opponents and the same number of games, and of course the same machine with all other conditions identical.

Say we do Hiarcs, which has a reputation for a very good book

- Hiarcs generic book vs 10 opponents with generic book, 50 games each
- Hiarcs own book vs 10 opponents with generic book, 50 games each

This isolates the effect of Hiarcs using it's own book instead of a generic. And then run the games through ELOstat or bayeselo and compute the ratings. Whether 500 games each option is enough so that error margins are small enough is debateable.

Would have to be blitz of course.

Any other ideas ? I'm willing to run such a test

Question is: Should the generic book have any move restriction, of 12 moves say ? CEGT and CCRL have such a restriction, so that kind of makes sense, but then again in this little test the Hiarcs "own book" would have no such restriction, so it also makes sense to have the generic book unrestricted as well...

You also need to test a weaker opponent under the same conditions. It is quite likely that a strong engine will see less gain from its "own book" than a much weaker engine that depends on the book to avoid problems it can't see on its own...

[Spock]

You also need to test a weaker opponent under the same conditions. It is quite likely that a strong engine will see less gain from its "own book" than a much weaker engine that depends on the book to avoid problems it can't see on its own...

Very good point

[Spock]

Hi Ray! Your experiment plan looks good. I think 500 games is not enough, I suggest at least 1000. For example using 20 opponents and 50 games per each. More opponents is always good, but those oponents should not be more than 200 points lower or higher than engine you test. IMO, some weaker engine would suit better because it will be easier to find enough opponents, both stronger and weaker than itself.

Yes - it will be difficult to find 20 opponents for Hiarcs. A weaker engine would be much better. Junior 10 is a much better choice then. It has a reputation for a very strong book, and I think I can find 10 opponents either side of it within a sensible ELO range

I don't think Junior 10 is what Bob Hyatt would have meant as a "weaker engine" but it's all I can think of. Unless of course it's an experiment I could also try with Crafty

[Sedat Canbaz]

Hello Ray,

No doubt that its will very useful and intersting test !

Keep up the good work,
Sedat

[Tony Thomas]

Hi Ray! Your experiment plan looks good. I think 500 games is not enough, I suggest at least 1000. For example using 20 opponents and 50 games per each. More opponents is always good, but those oponents should not be more than 200 points lower or higher than engine you test. IMO, some weaker engine would suit better because it will be easier to find enough opponents, both stronger and weaker than itself.

Yes - it will be difficult to find 20 opponents for Hiarcs. A weaker engine would be much better. Junior 10 is a much better choice then. It has a reputation for a very strong book, and I think I can find 10 opponents either side of it within a sensible ELO range

I don't think Junior 10 is what Bob Hyatt would have meant as a "weaker engine" but it's all I can think of. Unless of course it's an experiment I could also try with Crafty

I was going to try to convice you about Junior 10. Main reason being, Junior was able to tie Rybka at SSDF testing using own book and didnt do so well against others. The problem with trying a non-commercial program is that most of them dont have a famous booker, also there isnt people trying to take out bad lines from those books every minute of the day. Movei comes to mind, Uri built the book from Pgn using win ratio for black and white and a minimum amount of games per move. No personal tuning of Movei book as far as I know.

I agree with your choice.

[Peter Berger]

I think that what you want to test is nearly untestable in practice.

The commercially distributed "own books" are not tournament books like the ones you'd use in a tournament or match.
They have to be much wider - as no one would want a program that only plays one or two specific openings, even if it does it extremely well and in a sophisticated way . And such books would be *extremely* vulnerable against a learner anyway, if it isn't possible to tune and update them after each game.

As a book author you can do these tests, but to get a significant number you'd have to run a big number of games, including the (rather tedious) process of updating the book after each single game played.

In practice I think none of this has ever been tested too well, because the time needed for this is probably used for improving the book instead .

Peter

[hgm]

Following on a discussion in another thread - there seems to be quite a bit of interest in what an engine can gain by using it's own book, instead of the generic books used by all engines as in CEGT and CCRL

Isn't this already done? I remember to have seen a test, either here or on the Winboard forum, where 6 top engines with book and the same 6 top engines without book played a round-robin with a large number of games.

[Spock]

Following on a discussion in another thread - there seems to be quite a bit of interest in what an engine can gain by using it's own book, instead of the generic books used by all engines as in CEGT and CCRL

Isn't this already done? I remember to have seen a test, either here or on the Winboard forum, where 6 top engines with book and the same 6 top engines without book played a round-robin with a large number of games.

If it has been done already, that's good