Clop and cutechess

[sedicla]

First I'd like to say thank you so much to the authors of those two tools. Those are great resources you can have have to build your engine.

I started using clop to tune my engine, and I have no idea about the theory behind it, but I understand it works

Just to make sure I'm using correctly, I believe I have to pay attention to the plot tab having time in the X and the parameter on Y axis. An then at the Max tab to know which is tuned parameter value. Of course after couple of thousand games played. Should say 10000??

Also I like to share how I am running the games, I'm not using any script, clop is calling my engine directly and I modified to run the game against itself and print the result.
The modification is to select the color to tune (using the seed) and then modify the parameter being tuned accordingly. Color to tune will use values passed by clop, and the other color use the original parameters.

In this case I am disabling transposition table and evaluation/pawn caches, since it cannot have different scores in this game where eval will have different scores. I'm thinking I can have 2 tables in this case. Maybe i'll modify that later.


Anyway, thanks again to clop/cutechess guys and I appreciate any comments.

[jdart]

There are possible issues using your own engine as the reference opponent. You might want to consider tuning against a variety of other engines, since each will have its own strengths and weaknesses.

Re internally using the engine to self-play: I also have some code to enable this, but it uses two separate copies of the hashtable, so hash can be enabled w/o the problem of two different evals entering the same hashtable. (This is relatively easy to do in C++ because the hashtable and other global search variables are encapsulated in a class).

-Jon

[sedicla]

Yes I understand, but for now I don't have much computing power to use multiple engines. I will use for now self test and see how it goes. I expect some improvement since I haven't tuned many parameters.

And was thinking the same about hash table.

[nionita]

You might want to consider tuning against a variety of other engines, since each will have its own strengths and weaknesses.
-Jon

I was wondering if CLOP can be used in this scenario. I was thinking to play a gauntlet againgst a few different engines for every sample point CLOP wants to collect. Unfortunately CLOP expects only one of W, D or L, so I would have to define some limits for the result of the gauntlet to translate to the expected W/D/L, like:

Code: Select all

res <= low --> L
res > high --> W
otherwise  --> D

which first introduces two more or less artificially parameters, and second, it probably introduces a lot more noise.

So for now I also optimise against only one reference engine.

[jdart]

It sounds like you are trying to do by hand (and the hard way) what clop + cutechess-cli do already, if you set them up right. clop will run games with a whole bunch of variant parameters, and do correlation over the results. A script connects it to cutechess-cli which actually runs the games and returns the right result strings to clop. I think there is a sample python script in the distribution.

FYI this is the script I am using:

Code: Select all

#!/usr/bin/python
# -*- coding: utf-8 -*-

from subprocess import Popen, PIPE
import sys
#import exceptions


# Path to the cutechess-cli executable.
# On Windows this should point to cutechess-cli.exe
cutechess_cli_path = '/home/jdart/chess/cutechess-cli/cutechess-cli'

# The engine whose parameters will be optimized
engine = 'cmd=/home/jdart/chess/engines/arasan16.2/arasanx-64 proto=xboard option.Hash=256'

# Format for the commands that are sent to the engine to
# set the parameter values. When the command is sent,
# {name} will be replaced with the parameter name and {value}
# with the parameter value.
engine_param_cmd = 'option {name}={value}'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'name="Gaviota 0.86" cmd=/home/jdart/chess/engines/gaviota-0.86-linux/gaviota-0.86-linux64 dir=/home/jdart/chess/engines/gaviota-0.86-linux proto=xboard',
    'cmd=/home/jdart/chess/engines/crafty-23.5/crafty dir=/home/jdart/chess/engines/crafty-23.5 name="Crafty 23.5" proto=xboard option.Threads=1 option.Hash=256',
    'cmd=/home/jdart/chess/engines/EXchess_v7.03/EXchess_v7.03b_linux64 dir=/home/jdart/chess/engines/EXchess_v7.03 name="Exchess 7.03" proto=xboard option.Hash=256',
    'cmd=/home/jdart/chess/engines/arasan15.6/arasanx-64 dir=/home/jdart/chess/engines/arasan15.6 name="Arasan 15.6" proto=xboard option.Hash=256',
]

# Additional cutechess-cli options, eg. time control and opening book
#options = '-each tc=40/2+0.01 -draw 80 1 -resign 5 500'
#options = '-resign 3 500 -draw 20 5 -each tc=2+0.05 book=varied.bin'
options = '-resign movecount=3 score=500 -each tc=0:05+0.1 book=/home/jdart/chess/book/varied.bin'

def main(argv = None):
    if argv is None:
        argv = sys.argv[1:]

    if len(argv) == 0 or argv[0] == '--help':
        print(__doc__)
        return 0

    argv = argv[1:]
    if len(argv) < 3 or len(argv) % 2 == 0:
        print('Too few arguments')
        return 2

    clop_seed = 0
    try:
        clop_seed = int(argv[0])
    except exceptions.ValueError:
        print('Invalid seed value: %s' % argv[0])
        return 2

    fcp = engine
    scp = opponents[(clop_seed >> 1) % len(opponents)]

    # Parse the parameters that should be optimized
    for i in range(1, len(argv), 2):
        # Make sure the parameter value is numeric
        try:
            float(argv[i + 1])
        except exceptions.ValueError:
            print('Invalid value for parameter %s: %s' % (argv[i], argv[i + 1]))
            return 2
        # Pass CLOP's parameters to the engine by using
        # cutechess-cli's initialization string feature
        initstr = engine_param_cmd.format(name = argv[i], value = argv[i + 1])
        fcp += ' initstr="%s"' % initstr

    # Choose the engine's playing side (color) based on CLOP's seed
    if clop_seed % 2 != 0:
        fcp, scp = scp, fcp

    cutechess_args = '-srand %d -engine %s -engine %s %s' % (clop_seed >> 1, fcp, scp, options)
    command = '%s %s' % (cutechess_cli_path, cutechess_args)

    # print(command)

    # Run cutechess-cli and wait for it to finish
    process = Popen(command, shell = True, stdout = PIPE)
    output = process.communicate()[0]
    if process.returncode != 0:
        print('Could not execute command: %s' % command)
        return 2

    # Convert Cutechess-cli's result into W/L/D
    # Note that only one game should be played
    result = -1
    for line in output.splitlines():
        if line.startswith('Finished game'):
            if line.find(": 1-0") != -1:
                result = clop_seed % 2
            elif line.find(": 0-1") != -1:
                result = (clop_seed % 2) ^ 1
            elif line.find(": 1/2-1/2") != -1:
                result = 2
            else:
                print('The game did not terminate properly')
                return 2
            break

    if result == 0:
        print('W')
    elif result == 1:
        print('L')
    elif result == 2:
        print('D')

if __name__ == "__main__":
    sys.exit(main())

[nionita]

It sounds like you are trying to do by hand (and the hard way) what clop + cutechess-cli do already

No, of course, I wouldn't do it by hand. What I proposed was to start a gautlet with the those opponents. Then the test engine will run all the games of that tournament with the same parameters, producing one sample point, which is more consistent.

But this does not work because Clop expects a tree way result, W/D/L.

In your proposal you play with all that reference engines the same number of games, but the games are played with different parameters. Clop will probably weight the results the same way, which shouldn't be, as the reference engines have different strenghts and weaknesses and this is a source of noise.

[jdart]

There is noise anyway, and the correlation function smooths that out. The way I posted is how Stockfish is using it and most programs. It does work given enough games.

--Jon