Moderators: hgm, Rebel, chrisw
I do not understand exactly what this means.CLOP will not estimate the strength of each individual opponent, but it will maximize the winning rate over replications.
It does not matter if engines are of different strengths. If you have N opponents, and use N replications, this means that, for a given value of parameters, CLOP will play a game against each opponent. CLOP cannot choose the opponent it plays against. It plays against every opponent equally. So it should work well, regardless of their strengths. Of course, as always, you should make sure that the winning rate is not close to 0% or 100%.Michel wrote:I do not understand exactly what this means.CLOP will not estimate the strength of each individual opponent, but it will maximize the winning rate over replications.
Does this mean that there is a risk that CLOP will optimize for
winning against the weaker opponents?
If so the pool should probably only contain stronger engines....
Code: Select all
2677 1189 1027 0.536552 2 7 4 5 0
2678 1190 1027 0.536761 2 7 4 5 0
2679 1190 1028 0.536519 2 7 4 5 0
2680 1191 1028 0.536728 2 7 4 5 0
2681 1191 1029 0.536487 2 7 4 5 0
2682 1192 1029 0.536695 2 7 4 5 0
2683 1193 1029 0.536904 2 7 4 5 0
2684 1193 1030 0.536662 2 7 4 5 0
2685 1193 1031 0.536421 2 7 4 5 0
2686 1193 1031 0.536421 2 7 4 5 0
2687 1193 1031 0.536421 2 7 2 4 1
2688 1194 1031 0.536629 2 7 2 4 1
It's normal. Because quadratic regression is costly, CLOP does not update the regression after each sample. With 2000 samples, it updates the regression every 200 samples. Re-starting the experiment refreshed the regression, so this created a sudden change in estimated maximum.Michel wrote:Look at the next to last line. This is where the experiment was restarted after an engine crash.
"Central" is the win rate for all samples with w(x)=1.Michel wrote:I think I now understand everything except one thing. What is the meaning of the column "Central" in the "Win Rate" pane?
Code: Select all
#!/usr/bin/env python
#############################################################################
"""
DummyScript.py
This is an example script for use with parallel optimization. In order to
apply clop algorithms to your own problem, you should write a script that
behaves like this one.
Arguments are:
#1: processor id (symbolic name, typically a machine name to ssh to)
#2: seed (integer)
#3: parameter id of first parameter (symbolic name)
#4: value of first parameter (float)
#5: parameter id of second parameter (optional)
#6: value of second parameter (optional)
...
This script should write the game outcome to its output:
W = win
L = loss
D = draw
For instance:
$ ./DummyScript.py node-01 4 param 0.2
W
"""
#############################################################################
import sys
import math
import random
import time
import os
#
# Log script invocations
#
f = open('pawnval.log', 'a')
print(sys.argv, file=f)
#
# Print doc if not enough parameters
#
if len(sys.argv) < 5:
print(__doc__)
sys.exit()
#
# Sleep for a random amount of time
#
# random.seed(int(sys.argv[2]))
# time.sleep(random.random() * 2)
#
# Parse parameter values
#
i = 4
params = []
while i < len(sys.argv):
# params.append(float(sys.argv[i]))
params.append(int(sys.argv[i]))
i += 2
#
# Compute winning probability
#
# d2 = 0
# for i in range(len(params)):
# delta = params[i] - 0.3456789
# d2 += delta * delta * 10
# draw_elo = 100
# draw_rating = draw_elo * math.log(10.0) / 400.0
# win_p = 1.0 / (1.0 + math.exp(d2 + draw_rating))
# loss_p = 1.0 / (1.0 + math.exp(-d2 + draw_rating))
#
# Draw a random game according to this probability
#
# r = random.random()
# if r < loss_p:
# print("L")
# elif r > 1.0 - win_p:
# print("W")
# else:
# print("D")
# Run a game with param1=params[0]
fo = open("eychessu.ini","w")
fo.write("[param]\n");
fo.write("param1=" + str(params[0]) + "\n");
fo.write("param2=" + str(params[1]) + "\n");
fo.write("param3=" + str(params[2]) + "\n");
fo.write("param4=" + str(params[3]) + "\n");
fo.write("param5=" + str(params[4]) + "\n");
fo.close()
if os.path.isfile("92h-3dc0.CHK") :
os.remove("92h-3dc0.CHK");
if os.path.isfile("3dc-92h0.CHK") :
os.remove("3dc-92h0.CHK");
os.system("uccileag.exe < ucci3dc.in > ucci3dc.out");
score = 0
roundnum = 0
for line in open("ucci3dc.out"):
parts = line.split(' ')
# print len(parts)
x = 0
while x+1<len(parts) :
parts[x]=parts[x].strip()
# print(parts[x])
if roundnum == 0 :
if parts[x] == '1/2-1/2' :
score = 1
roundnum = 1
elif parts[x] == '0-1' :
score = 0
roundnum = 1
elif parts[x] == '1-0' :
score = 2
roundnum = 1
elif roundnum == 1 :
if parts[x] == '1/2-1/2' :
score = score + 1
roundnum = 2
elif parts[x] == '0-1' :
score = score + 2
roundnum = 2
elif parts[x] == '1-0' :
# score = score + 0
roundnum = 2
elif parts[x] == 'Final' :
if score == 2 :
print("D")
if score < 2 :
print("L")
if score > 2 :
print("W")
x=x+1
os.rename("ucci3dc.out", "ucci3dc.out"+sys.argv[2])
# print("roundnum=",roundnum)
# print("score=",score) Code: Select all
#
# pawnval.clop
#
# Example of experiment definition
#
# Name (used for .log and .dat files)
Name pawnval
# Script for running a game. See DummyScript.py for details.
Script c:/python32/python.exe pawnval.py
# Parameter(s) to be optimized
# <parameter_type> <name> <min> <max>
# <parameter_type> may be:
# LinearParameter
# IntegerParameter
# GammaParameter
# IntegerGammaParameter
# For GammaParameter, quadratic regression is performed on log(x)
# Warning: 123 and not 123.0 should be used for IntegerParameter
# int g_param1 = 24; //VP default 24
# int g_param2 = 95; //VPR default 95
# int g_param3 = 32; //ENDPAWNBONUS default 32
# int g_param4 = 32; //FUTPAWN default 32
# int g_param5 = 28; //ENDFUTPAWN default 28
IntegerParameter vp 20 40
IntegerParameter vpr 90 100
IntegerParameter epb 20 40
IntegerParameter fp 20 40
IntegerParameter efp 20 40
# LinearParameter p2 0.0 1.0
# LinearParameter p3 0.0 1.0
# LinearParameter p4 0.0 1.0
# LinearParameter p5 0.0 1.0
# This could be the list of machine names in a distributed experiment.
# In order to run 4 games in parallel, 2 on machine1, 2 on machine2:
Processor machine1
# Processor machine1
# Processor machine2
# Processor machine2
# Call the script "Replications" times in a row with identical parameter values
# Replications may be used to alternate colors, for instance.
# Seed % Replications would indicate color.
# Replications 2
# Parameters of statistical model of outcome
# For binary outcome (Win/Loss, no draws), use "DrawElo 0"
# For chess, use "DrawElo 100"
DrawElo 100
# Regression parameters
# H 3 is recommended (it is the default value)
# Correlations may be "all" (default) or "none"
# Even if variables are not correlated "all" should work well. The problem is
# that the regression might become very costly if the number of variables is
# high. So use "Correlations none" only if you are certain parameters are
# independent or you have so many variables that "all" is too costly.
H 3
Correlations all
Hi Remi,Rémi Coulom wrote:Thanks Edward for your kind words. I am glad that programmers enjoy using CLOP.
I took a quick look at your script, and it seems you are not using CLOP optimally. If I understand correctly, each invocation of your script runs two games, and you write a W/L/D result based on the total outcome of these two games.
It would be better to let CLOP know about the detailed outcomes of these two games. If you wish to play two games for some parameters, one with White, the other with Black, then you should rather use the "Replications" parameter of CLOP. Then, your script would play a game with White or Black, depending on the parity of the seed.
Rémi
Code: Select all
Round 1:
Eychessu_1.892h 1-0 3DChess092 (92h-3dc0.PGN)
Rank List after Round 1:
No Abbr Engine-Name ELO K Win Draw Loss Score
=======================================================
1 92h Eychessu_1.892h 2010 20 1 0 0 1
2 3dc 3DChess092 1990 20 0 0 1 0
=======================================================
Round 2:
3DChess092 1/2-1/2 Eychessu_1.892h (3dc-92h0.PGN)
Rank List after Round 2:
No Abbr Engine-Name ELO K Win Draw Loss Score
=======================================================
1 92h Eychessu_1.892h 2010 20 1 1 0 1.5
2 3dc 3DChess092 1990 20 0 1 1 0.5
=======================================================
=== End of Match Process ===
Final Result:
No Abbr Engine-Name ELO K Win Draw Loss Score
=======================================================
1 92h Eychessu_1.892h 2010 20 1 1 0 1.5
2 3dc 3DChess092 1990 20 0 1 1 0.5
=======================================================