ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

tuning for the uninformed
Goto page 1, 2, 3, 4  Next
 
Post new topic       TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Threaded
View previous topic :: View next topic  
Author Message
Folkert van Heusden



Joined: 14 Mar 2008
Posts: 890
Location: Gouda, the Netherlands

PostPosted: Thu Nov 23, 2017 10:37 am    Post subject: tuning for the uninformed Reply to topic Reply with quote

Hi,

What about the following method for tuning evaluation:

- run through, say, 1000 positions
- generate a (semi-)random set of tuning parameters
- get their eval from your program
- get the eval from the must-be-good program (I'm using stockfish)
- compare those 1000 pairs by using https://en.wikipedia.org/wiki/Pearson_correlation_coefficient
- coefficient > previous_coefficient? then remember this tuning parameters set

What do you think?
_________________
https://vanheusden.com/feeks/
https://vanheusden.com/Embla/
https://vanheusden.com/DeepBrutePos/
https://vanheusden.com/pos/
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
Henk van den Belt



Joined: 27 May 2013
Posts: 4972

PostPosted: Thu Nov 23, 2017 11:49 am    Post subject: Re: tuning for the uninformed Reply to topic Reply with quote

flok wrote:
Hi,

What about the following method for tuning evaluation:

- run through, say, 1000 positions
- generate a (semi-)random set of tuning parameters
- get their eval from your program
- get the eval from the must-be-good program (I'm using stockfish)
- compare those 1000 pairs by using https://en.wikipedia.org/wiki/Pearson_correlation_coefficient
- coefficient > previous_coefficient? then remember this tuning parameters set

What do you think?


Something is better then nothing.

Are these 1000 positions representative for every position that may occur when playing games. Probably not.

Don't you want to create something different than Stockfish. I already have a copy of Stockfish running on my machine.

But something is better than nothing. (Skipper is nothing). Or not. For instance. Why waste time on something which won't make it. Maybe only for generating better ideas.


Last edited by Henk van den Belt on Thu Nov 23, 2017 11:52 am; edited 2 times in total
Back to top
View user's profile Send private message
Mehdi Amini



Joined: 05 Jun 2014
Posts: 105
Location: Iran

PostPosted: Thu Nov 23, 2017 11:49 am    Post subject: Re: tuning for the uninformed Reply to topic Reply with quote

flok wrote:
Hi,

What about the following method for tuning evaluation:

- run through, say, 1000 positions
- generate a (semi-)random set of tuning parameters
- get their eval from your program
- get the eval from the must-be-good program (I'm using stockfish)
- compare those 1000 pairs by using https://en.wikipedia.org/wiki/Pearson_correlation_coefficient
- coefficient > previous_coefficient? then remember this tuning parameters set

What do you think?


You can use Genetic Algorithm too. See this link for instance:

https://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol1/hmw/article1.html
_________________
www.my-c-codes.com/

Farewell.
Back to top
View user's profile Send private message Visit poster's website
Henk van den Belt



Joined: 27 May 2013
Posts: 4972

PostPosted: Thu Nov 23, 2017 11:53 am    Post subject: Re: tuning for the uninformed Reply to topic Reply with quote

Yes he is doing hill climbing now or perhaps only sampling.
Back to top
View user's profile Send private message
Folkert van Heusden



Joined: 14 Mar 2008
Posts: 890
Location: Gouda, the Netherlands

PostPosted: Thu Nov 23, 2017 12:26 pm    Post subject: Re: tuning for the uninformed Reply to topic Reply with quote

Only sampling.
_________________
https://vanheusden.com/feeks/
https://vanheusden.com/Embla/
https://vanheusden.com/DeepBrutePos/
https://vanheusden.com/pos/
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
Henk van den Belt



Joined: 27 May 2013
Posts: 4972

PostPosted: Thu Nov 23, 2017 12:44 pm    Post subject: Re: tuning for the uninformed Reply to topic Reply with quote

Sampling is ok for tuning two parameters or so. Otherwise it is terribly slow.
O wait if you tune it badly it generalizes better. So it will do better on evaluating unseen positions. Somewhere there is an optimum between bad tuning and 'overtuning'.

Other constraint is that tuning should not cost too much time so better use hill climbing with restart. Or genetic algorithm (with restart?)
Back to top
View user's profile Send private message
Alexandru Mosoi



Joined: 16 Jan 2015
Posts: 426

PostPosted: Thu Nov 23, 2017 2:07 pm    Post subject: Re: tuning for the uninformed Reply to topic Reply with quote

I tried pearson correlation in the past, but it's not a good measure. Right now my experiments with evolving the eval parameters use texel tuning on a set of 100K positions (so I can train ~6000 models per day).

In terms of the error rate, I got very close to the hand tuned model (stable version). If you want to see the fully automatic trained eval with almost 0 human intervention check [1] or [2]. The evolved version is 100 Elo weaker than the stable version of Zurichess. I need to add back the pawns cache, though.


[1] https://bitbucket.org/brtzsnr/zurichess/src/c37cb071903537a543d76607b75ce5d43c192710/engine/eval.go?at=evolve&fileviewer=file-view-default
[2] https://bitbucket.org/brtzsnr/zurichess/src/59a02f5ae8b725e6d5442f5ddbb12a439b6381bf/engine/eval.go?at=lmr1&fileviewer=file-view-default
_________________
zurichess - http://www.zurichess.xyz
Back to top
View user's profile Send private message Visit poster's website
Folkert van Heusden



Joined: 14 Mar 2008
Posts: 890
Location: Gouda, the Netherlands

PostPosted: Thu Nov 23, 2017 2:16 pm    Post subject: Re: tuning for the uninformed Reply to topic Reply with quote

Yeah I looked at texel tuning but it looked rather complicated. This pearson was implemented in an hour during the morning commute Very Happy
_________________
https://vanheusden.com/feeks/
https://vanheusden.com/Embla/
https://vanheusden.com/DeepBrutePos/
https://vanheusden.com/pos/
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
Folkert van Heusden



Joined: 14 Mar 2008
Posts: 890
Location: Gouda, the Netherlands

PostPosted: Thu Nov 23, 2017 5:34 pm    Post subject: Re: tuning for the uninformed Reply to topic Reply with quote

If anyone is willing to explain the Texel tuning method tht would be great!

Sofar I understand I have to let it play (well, run QS + eval on FENs) millions of games and then do something with the evaluation-value. But what? I don't understand the wiki explanation.
_________________
https://vanheusden.com/feeks/
https://vanheusden.com/Embla/
https://vanheusden.com/DeepBrutePos/
https://vanheusden.com/pos/
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
Sander Maassen vd Brink



Joined: 28 Jan 2017
Posts: 104
Location: The Netherlands

PostPosted: Thu Nov 23, 2017 6:47 pm    Post subject: Re: tuning for the uninformed Reply to topic Reply with quote

flok wrote:
If anyone is willing to explain the Texel tuning method tht would be great!

Sofar I understand I have to let it play (well, run QS + eval on FENs) millions of games and then do something with the evaluation-value. But what? I don't understand the wiki explanation.


The basic idea is pretty simple: calculate the error of the evaluation when it is compared to the actual outcome of the positions. Lower a particular evaluation parameter and check if the error has improved, if not, higher the parameter, if again not improved, keep the original value. Do this for all parameters until you have reached the lowest error.
_________________
https://github.com/sandermvdb/chess22k
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic       TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions All times are GMT
Goto page 1, 2, 3, 4  Next
Threaded
Page 1 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads