TalkChess.com :: View topic - What your opinion about this testing methodology?

TalkChess.com
Hosted by Your Move Chess & Games

FAQ

Memberlist

Usergroups

Profile

What your opinion about this testing methodology?

TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

View previous topic :: View next topic

Author

Message

Robert Hyatt

Joined: 27 Feb 2006
Posts: 15820
Location: Birmingham, AL

Post subject: Re: What your opinion about this testing methodology? Posted: Wed Apr 18, 2012 7:28 pm

Sven Schüle wrote:

Kempelen wrote:

Ferdy wrote:

Your methodology is favorable considering your goal. There is no point testing 1. b3 e5 positions if it is not in your engines repertoire. Only use test positions where you want your engine to be. Of course there are drawbacks, but that can be overcomed as you say using large number of selected positions.

Perhaps start from smaller number of positions, and as your engine able to improve from it, then add other positions, to be considered in its repertoire.

But I have a bad feeling about it, to me the engine should be able to handle all positions, it can be blocked, open, full of pinned pieces, etc.

I think you misunderstand my idea. The goal is not test only a limited set of opening positions, but a large and varied set of starting middle-game positions. The point is repeating always the same games with the same positions, but enought positions to say the engine is played a varied.

The point is, the positions are selected once by random but then always the same positions are used for testing. That's exactly what Bob is doing for a long while now, and also lots of other people, so it is not a new method but kind of "de facto standard". I recall there were long discussions about the details few years ago. Doing it that way instead of newly choosing different positions by random each time has been found to result in lower error bars as far as I remember. I guess Bob and the other experts in statistics can explain the exact reasons.

Sven

It really doesn't influence the BayesElo error bar, obviously, but if you choose random openings, you introduce one more variable, as one run might choose openings you play well, and the next uses openings you play poorly, producing a false impression.

Display posts from previous:

Subject	Author	Date/Time
What your opinion about this testing methodology?	Fermin Serrano	Tue Apr 17, 2012 5:28 pm
Re: What your opinion about this testing methodology?	Ferdinand Mosca	Wed Apr 18, 2012 6:45 am
Re: What your opinion about this testing methodology?	Fermin Serrano	Wed Apr 18, 2012 7:56 am
Re: What your opinion about this testing methodology?	Sven Schüle	Wed Apr 18, 2012 9:40 am
Re: What your opinion about this testing methodology?	Lucas Braesch	Wed Apr 18, 2012 11:41 am
Re: What your opinion about this testing methodology?	Sven Schüle	Wed Apr 18, 2012 11:52 am
Re: What your opinion about this testing methodology?	Robert Hyatt	Wed Apr 18, 2012 7:28 pm
Re: What your opinion about this testing methodology?	Ferdinand Mosca	Wed Apr 18, 2012 10:12 am
Re: What your opinion about this testing methodology?	Vincent Diepeveen	Wed Apr 18, 2012 11:51 am
Re: What your opinion about this testing methodology?	Mincho Georgiev	Wed Apr 18, 2012 7:52 am
Re: What your opinion about this testing methodology?	Mincho Georgiev	Wed Apr 18, 2012 11:46 am
Re: What your opinion about this testing methodology?	Don Dailey	Wed Apr 18, 2012 5:05 pm
Re: What your opinion about this testing methodology?	Robert Hyatt	Wed Apr 18, 2012 7:27 pm

TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum