cutechess-cli in python

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Daniel Shawul
Posts: 3914
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: cutechess-cli in python

Post by Daniel Shawul » Fri Jun 24, 2016 3:29 am

I got one... :-)

Code: Select all

        import ctypes
        pittichess-noncli = cdll.LoadLibrary("cutechesslib.so")
Seriously though, once you write your backend library in C/C++, then writing a frontend for many languages (scripting and otherwise) is a piece of cake. For example, we use this approach to ease programming the vast array of many-core architecture.

mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 7:17 pm

Re: cutechess-cli in python

Post by mcostalba » Fri Jun 24, 2016 7:26 am

I don't understand why all these negative comments for this new nice project.

If Lucas wants to write an engine manager what's the problem?

If he wants to write it in Phyton what's the problem?

I am not good at Python but I really like it, I have developed some small stuff for fishtest testing framework and I found it extremely pleasant to work with. The only serious issue (in part due to my reduced experience) is scalability with program size. When sources get complex you really start to miss the strict type checking and any addition or change can break the code in difficult to find ways (I think this is the reason big Python projects have an almost as big part of code devoted to test suites, this is a workaround to missing type check).

I encourage Lucas to not give up: starting a project is the easy part :-)

mar
Posts: 2122
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: cutechess-cli in python

Post by mar » Fri Jun 24, 2016 7:32 am

lucasart wrote:Have you measured it ? Have you even tried it ? (+same questions on PyPy).
I have measured performance of several interpreted scripting languages. Python really didn't shine. Pypy actually impressed me, even if not competitive with LuaJIT (which is absolutely fantastic).
I claim that you won't see a difference in any realistic testing scenario, because almost all the time is spend waiting for I/O operations.
That remains to be seen, what about GC? Are you sure it won't trigger often during the test run? Are you sure it won't produce additional noise?
Talk is cheap, show us your code...
Didn't like the feedback? Quoting Linus instead of fixing a bug is cheap.
You know very well the hack doesn't prevent the race.

I don't see any point in trying to mimic something that works very well, that was your choice. I have better things to do.

mar
Posts: 2122
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: cutechess-cli in python

Post by mar » Fri Jun 24, 2016 9:16 am

AndrewGrant wrote:Hey, I'm proud of that "nasty" hack! :)
Well, you shouldn't be :)
I see a problem with the API where it throws an exception rather than returning null if queue is empty.
However, since you're in a try-except block anyway, why not jobQueue.get(False) and handle the Queue.Empty exception?
If I'm not mistaken you fill the queue with all games to be played in advance.

mar
Posts: 2122
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: cutechess-cli in python

Post by mar » Fri Jun 24, 2016 9:34 am

mar wrote:You know very well the hack doesn't prevent the race.
Correction: now I see that the hack actually works in all cases so no race/no hanging is possible

AndrewGrant
Posts: 583
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: cutechess-cli in python

Post by AndrewGrant » Fri Jun 24, 2016 2:43 pm

To restate. I'm proud of my nasty hack.

mar
Posts: 2122
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: cutechess-cli in python

Post by mar » Fri Jun 24, 2016 6:57 pm

AndrewGrant wrote:To restate. I'm proud of my nasty hack.
Good for you. And good to know who defecated Lucas' code :D

AndrewGrant
Posts: 583
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: cutechess-cli in python

Post by AndrewGrant » Fri Jun 24, 2016 8:33 pm

I have measured performance of several interpreted scripting languages. Python really didn't shine. Pypy actually impressed me, even if not competitive with LuaJIT (which is absolutely fantastic).
You understand that python is not actually managing the running engines, right? Once the engines are launched and given commands they don't give a damn about the python code that spawned them. The only control that python has over them is knowing their process IDs and being able to kill them when python wants to.

So for performance are you worried about the time between sending and receiving commands? Are you suggesting we should be worried about the millisecond it takes to give the engine the command, get the move, and update the board? Are you afraid of wasting a few milliseconds a game? If the answer is yes, you better launch a cutechess-x86-assembly project of your own.
That remains to be seen, what about GC? Are you sure it won't trigger often during the test run? Are you sure it won't produce additional noise?


Yes, and Yes.

My tests showed an average of less than 1 garbage collection cycle per game, taking, on average, less than 1 millisecond, at most 3 milliseconds.

In thousands of games across multiple cores, a garbage collection cycle was never run in such a way that it affected the elapsed Time calculation for an engine. Which, is the expected result, because despite the mysticism behind garbage collection, it is not entirely unpredictable.
Didn't like the feedback? Quoting Linus instead of fixing a bug is cheap.
You know very well the hack doesn't prevent the race.
Correction: now I see that the hack actually works in all cases so no race/no hanging is possible
After you took some time to think about it you came around to see that it worked. If you don't like the response Lucas gave, then you SHOULD write some code to show us. I tried a dozen things, and found this is be the most simple, reliable solution I could come up with. If you know of a better method, please share. I am always open to learning new things.

AndrewGrant
Posts: 583
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: cutechess-cli in python

Post by AndrewGrant » Fri Jun 24, 2016 9:17 pm

the queue.get(false) with exception handling is nice, but what if we want to fill the queue with games for multiple pairings? We have engines A, B, and C. We want A to play 25 vs B, B to play 25vs C, and C to play 25 vs A. We let each pairing play out their 25 games, before closing down those Engine pairings. There is more to it than you looked for.

User avatar
lucasart
Posts: 3104
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: cutechess-cli in python

Post by lucasart » Sat Jun 25, 2016 2:40 am

mar wrote: what about GC? Are you sure it won't trigger often during the test run? Are you sure it won't produce additional noise?
There's no GC in CPython, which uses reference counting. This is essentially equivalent to C++ RAII, where the object are destroyed when no longer needed (goes out of scope in C++, reference count drops to zero in CPython).

It's actually PyPy that uses GC. So I am surprised that you are now using GC as a last attempt argument to defend your position, because you said earlier that PyPy was faster than CPython (with of course no evidence to support this claim).

I recommend you learn a bit more about Python before crapping on it.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

Post Reply