Engine testing with opening books - why?

Discussion of computer chess matches and engine tournaments.

Moderator: Ras

pichy
Posts: 2564
Joined: Thu Mar 09, 2006 3:04 am

Re: Engine testing with opening books - why?

Post by pichy »

Dr.Wael Deeb wrote:
M ANSARI wrote:Yes, if you were to use as a standard one set of opening positions, you will encourage an engine make to tune his engine to that particular set which could not give you true insight of engine strength. Best is to use many different opening books from reputable authors and truncate the books at 7 moves or 10 moves or even 20 moves.

Personally I also like to look at the games played to see if I can identify any certain weakness or strength. For example after looking at many losses of N4, it is clear to me that the next generation of N4 will be quite a bit stronger since N4 seems to have a weakness that can quickly be fixed. N4 does not have accurate assessment of when connected pawns are stronger than pieces ... at least not as accurate as R3. That should not be too difficult to fix, but at the moment it is causing many losses to R3.
Don't know Majd,but I think that Naum is the engine making the biggest leaps toward Rybka and could be a potential candidate for knocking it down in the rating lists 8-)
I doubt it, but NOT using your opening book :roll:
User avatar
Dr.Wael Deeb
Posts: 9773
Joined: Wed Mar 08, 2006 8:44 pm
Location: Amman,Jordan

Re: Engine testing with opening books - why?

Post by Dr.Wael Deeb »

pichy wrote:
Dr.Wael Deeb wrote:
M ANSARI wrote:Yes, if you were to use as a standard one set of opening positions, you will encourage an engine make to tune his engine to that particular set which could not give you true insight of engine strength. Best is to use many different opening books from reputable authors and truncate the books at 7 moves or 10 moves or even 20 moves.

Personally I also like to look at the games played to see if I can identify any certain weakness or strength. For example after looking at many losses of N4, it is clear to me that the next generation of N4 will be quite a bit stronger since N4 seems to have a weakness that can quickly be fixed. N4 does not have accurate assessment of when connected pawns are stronger than pieces ... at least not as accurate as R3. That should not be too difficult to fix, but at the moment it is causing many losses to R3.
Don't know Majd,but I think that Naum is the engine making the biggest leaps toward Rybka and could be a potential candidate for knocking it down in the rating lists 8-)
I doubt it, but NOT using your opening book :roll:
What opening book you are talking about and rolling your stupid eyes :!: :?:
I don't have opening book for Naum so try to think before you open your mouth and talk nonsense....
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….
jdart
Posts: 4420
Joined: Fri Mar 10, 2006 5:23 am
Location: http://www.arasanchess.org

Re: Engine testing with opening books - why?

Post by jdart »

Those are all valid points, but as an engine author I much prefer to see tests with my own engine book.

First of all I put a lot of effort into it and I've tried to avoid bad lines. But if I find one in the book (or a tester does) I want to fix it.

Also most of my users are not running the engine under these conditions. They're playing against it with its own book or using it online. So testing with "own book" mimics what most consumers of the engine are doing with it. (Chess geeks do other things with it of course).
BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 5:18 am

Re: Engine testing with opening books - why?

Post by BubbaTough »

jdart wrote:Those are all valid points, but as an engine author I much prefer to see tests with my own engine book.

First of all I put a lot of effort into it and I've tried to avoid bad lines. But if I find one in the book (or a tester does) I want to fix it.

Also most of my users are not running the engine under these conditions. They're playing against it with its own book or using it online. So testing with "own book" mimics what most consumers of the engine are doing with it. (Chess geeks do other things with it of course).
Agreed.

-Sam
Spock

Re: Engine testing with opening books - why?

Post by Spock »

jdart wrote:Those are all valid points, but as an engine author I much prefer to see tests with my own engine book.

First of all I put a lot of effort into it and I've tried to avoid bad lines. But if I find one in the book (or a tester does) I want to fix it.

Also most of my users are not running the engine under these conditions. They're playing against it with its own book or using it online. So testing with "own book" mimics what most consumers of the engine are doing with it. (Chess geeks do other things with it of course).
Agreed also
User avatar
Graham Banks
Posts: 45228
Joined: Sun Feb 26, 2006 10:52 am
Location: Auckland, NZ

Re: Engine testing with opening books - why?

Post by Graham Banks »

Dr.Wael Deeb wrote: I use generic books only for the bookless engines,because........Sorry,I have to go now,I see Graham running toward me with an axe in his hand :shock:
Running for life regards,
Dr.D
Image
gbanksnz at gmail.com
User avatar
Dr.Wael Deeb
Posts: 9773
Joined: Wed Mar 08, 2006 8:44 pm
Location: Amman,Jordan

Re: Engine testing with opening books - why?

Post by Dr.Wael Deeb »

Graham Banks wrote:
Dr.Wael Deeb wrote: I use generic books only for the bookless engines,because........Sorry,I have to go now,I see Graham running toward me with an axe in his hand :shock:
Running for life regards,
Dr.D
Image
:lol:

:P
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….
pichy
Posts: 2564
Joined: Thu Mar 09, 2006 3:04 am

Re: Engine testing with opening books - why?

Post by pichy »

Dr.Wael Deeb wrote:
pichy wrote:
Dr.Wael Deeb wrote:
M ANSARI wrote:Yes, if you were to use as a standard one set of opening positions, you will encourage an engine make to tune his engine to that particular set which could not give you true insight of engine strength. Best is to use many different opening books from reputable authors and truncate the books at 7 moves or 10 moves or even 20 moves.

Personally I also like to look at the games played to see if I can identify any certain weakness or strength. For example after looking at many losses of N4, it is clear to me that the next generation of N4 will be quite a bit stronger since N4 seems to have a weakness that can quickly be fixed. N4 does not have accurate assessment of when connected pawns are stronger than pieces ... at least not as accurate as R3. That should not be too difficult to fix, but at the moment it is causing many losses to R3.
Don't know Majd,but I think that Naum is the engine making the biggest leaps toward Rybka and could be a potential candidate for knocking it down in the rating lists 8-)
I doubt it, but NOT if you use your GREAT opening book :roll:
What opening book you are talking about and rolling your stupid eyes :!: :?:
I don't have opening book for Naum so try to think before you open your mouth and talk nonsense....
I better NOT joke with you anymore, since you don't know how to take it. Sorry,I have to go now, I believe that you will send me private emails with viruses :lol:
User avatar
Dr.Wael Deeb
Posts: 9773
Joined: Wed Mar 08, 2006 8:44 pm
Location: Amman,Jordan

Re: Engine testing with opening books - why?

Post by Dr.Wael Deeb »

pichy wrote:
Dr.Wael Deeb wrote:
pichy wrote:
Dr.Wael Deeb wrote:
M ANSARI wrote:Yes, if you were to use as a standard one set of opening positions, you will encourage an engine make to tune his engine to that particular set which could not give you true insight of engine strength. Best is to use many different opening books from reputable authors and truncate the books at 7 moves or 10 moves or even 20 moves.

Personally I also like to look at the games played to see if I can identify any certain weakness or strength. For example after looking at many losses of N4, it is clear to me that the next generation of N4 will be quite a bit stronger since N4 seems to have a weakness that can quickly be fixed. N4 does not have accurate assessment of when connected pawns are stronger than pieces ... at least not as accurate as R3. That should not be too difficult to fix, but at the moment it is causing many losses to R3.
Don't know Majd,but I think that Naum is the engine making the biggest leaps toward Rybka and could be a potential candidate for knocking it down in the rating lists 8-)
I doubt it, but NOT if you use your GREAT opening book :roll:
What opening book you are talking about and rolling your stupid eyes :!: :?:
I don't have opening book for Naum so try to think before you open your mouth and talk nonsense....
I better NOT joke with you anymore, since you don't know how to take it. Sorry,I have to go now, I believe that you will send me private emails with viruses :lol:
No,I don't have such creatures in my system 8-)
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….
User avatar
Kirill Kryukov
Posts: 518
Joined: Sun Mar 19, 2006 4:12 am
Full name: Kirill Kryukov

Re: Engine testing with opening books - why?

Post by Kirill Kryukov »

jdart wrote:Those are all valid points, but as an engine author I much prefer to see tests with my own engine book.

First of all I put a lot of effort into it and I've tried to avoid bad lines. But if I find one in the book (or a tester does) I want to fix it.

Also most of my users are not running the engine under these conditions. They're playing against it with its own book or using it online. So testing with "own book" mimics what most consumers of the engine are doing with it. (Chess geeks do other things with it of course).
You are concerned about your own engine, a tester has to be concerned about all engines equally. You say that your engine is mostly used for playing with it. I think that out of all people using chess engines, most are doing analysis rather than playing with them.

You put a lot of effort into the book, so you prefer tests with own book. My guess is that authors that don't have the time, skill or interest to work on opening book will prefer tests without own books.

Still your view has right to exist of coruse.

Best,
Kirill